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Abstract 


In real distributed systems, processes may have only inexact information about the 
amount of real time needed for primitive operations such as process steps. This thesis 
studies the effect of this timing uncertainty on the real-time behavior of distributed systems. 
We consider a semi-synchronous model in which the amount of real time between process 
steps is known to be in the interval [c, cg] and every message is known to be delivered within 


time d of when it is sent. We use C = cg/c; as a measure of the timing uncertainty. 


We first study the problem of reaching agreement in the presence of failures. A simple 
argument derived from the case of synchronous processes shows that at least time (f + 1)d 
is required to tolerate f failures, while time (f + 1)Cd is sufficient to tolerate f stopping or 
omission failures by directly simulating the rounds of any synchronous consensus algorithm. 
We narrow this gap for omission failures, building on the nearly optimal algorithm of Attiya, 
Dwork, Lynch, and Stockmeyer which tolerates only stopping failures. If fewer than half the 
processes are faulty (n > 2f + 1), then the running time of our algorithm is 4(f + l)d+ 
Cd, which is within a factor of 4 of optimal and has minimal dependency on the timing 
uncertainty factor C’. If more than half the processes are faulty, then a more complicated 
analysis shows the running time is increased by approximately a factor of min(&, VC). We 
also present a general simulation for n > 3f +1 tolerant of Byzantine failures that simulates 
any synchronous algorithm at a cost of time 2C'd +4 d per round. 


Finally, motivated by the message inefficiency of our consensus algorithm for omission 
failures, we define a more realistic model of message links by limiting their capacity. If 
messages are sent too frequently on these message links, they may incur delay greater 
than d. For message links with capacity 4, we prove nearly tight upper and lower bounds of 
min(2Cd + d, C?d/p + Cd-+d) and min(2Cd + d/p, C?d/p + Cd +d) respectively for the 


time needed to detect stopping failures. 
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Chapter 1 


Introduction 


In real distributed systems, processes are likely to be neither perfectly synchronous nor com- 
pletely asynchronous. Many systems lie somewhere between these two extremes and can thus 
be more accurately modeled by a semi-synchronous model in which processes have inexact 
knowledge about real time. In our model, the degree of asychrony is captured by a parameter 
which we call the processes’ timing uncertainty. We will be particularly interested in how 
the magnitude of timing uncertainty affects the time complexity of distributed computing 
problems. In particular, we study the the time needed to reach consensus in the presence 
of omission failures and in the presence of Byzantine failures. We also introduce a model 
of message links with bounded-capacity and study the time needed to detect failures in a 


system using these message links. 


In a synchronous system, processors have perfectly synchronized clocks and distributed 
algorithms are often broken up into rounds of communication. In a single round of com- 
munication, each processor may receive messages from other processors, perform some local 
computation, and then send messages to other processors. The time required to perform 
local operations is generally assumed to be negligible and the time complexity of algorithms 
is therefore measured by the number of rounds of communication required. In an asyn- 
chronous system, the delay of messages is arbitrary and unbounded (or the relative rates 
of different processors are unbounded). The time complexity of an asynchronous algorithm 


is usually measured by letting one time unit equal the maximum delay of any message 


([Gal82, Awe85]). 


The model we use is a slightly simplified version of the semi-synchronous model intro- 
duced in [AL89], which is in turn based on the formal model of timed automata in [MMT90]. 
In this model, processors have inexact knowledge about the time needed to perform certain 
primitive operations. The model is formally described in Section 2.1, but is very simple: 


every message is delivered within time d of when it is sent and the amount of time between 


any two consecutive steps of any process is in the interval [c¢,,¢2]. Because process steps are 
the only events for which there is a lower bound, a process can deduce a lower bound on 
the amount of time for any interval of events only by counting the number of steps it takes 
in that interval. For instance, to ensure that time d elapses over an interval of events, a 
processor must count d/c; of its local steps, after these events it knows that at least time 
c,:d/c, = d (and at most c2-d/c,) has elapsed. We will be particularly interested in how 
this timing uncertainty factor of c2/c,, henceforth denoted C, affects the time complexity of 


problems relative to their synchronous time complexity. 


Of particular interest are problems that are intractable in an asynchronous setting yet 
have solutions with tight bounds in the synchronous setting. A simple example is the basic 
task of detecting the failure of stopped processes. Clearly, if there is no bound on message 
delay or relative process step time, then failures can never be detected with certainty; in a 
synchronous system, any stopping failure can be detected within approximately the maxi- 
mum message delay time. Another natural candidate is the consensus problem. It is well 
known that a completely asynchronous algorithm for consensus cannot tolerate the failure 
of even one process, whereas exactly f +1 rounds of synchronous communication are needed 


to tolerate f failures in a synchronous system. 


1.1 Reaching consensus—known time bounds 


The problem of reaching consensus in the presence of failures is one of the most well-studied 
problems in distributed computing. We consider the version of this problem for a system 
of n deterministic processes some f of which may fail, completely connected by a reliable 
message system. The processes begin executing at the same time, each with a private binary 
input, and must each decide on a binary value such that no two nonfaulty processes decide 
differently and if all processes begin with value v then v is the decision of all nonfaulty 
processes. In this thesis, we consider two kinds of process failure: send-omission failures, by 
which a process may unwittingly omit messages of an algorithm, and Byzantine failures, by 


which a process may exhibit arbitrary behavior. 


It is well known ({[FLP85]) that in an asynchronous system, this problem cannot be solved 
deteministically even if the only failure to be tolerated is the unannounced halting (stopping) 
of a single process. The work of [DDS87] methodically explores the synchrony necessary to 
reach consensus; they show that if there is no upper bound on message delay or there is no 
upper bound on the relative rate of process steps—if any of our bounds d, c,, or cz does not 


hold—then there is no deterministic solution tolerating even a single stopping failure. 


The time complexity of the consensus problem has been well studied in the synchronous 


rounds model (see, for example, [LSP82, PSL80, FL82, DS83, DLM82]). It is well known 
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that f+1 rounds of communication are both sufficient ([PSL80]) and necessary ([FL82, M85, 
DM86, CD86]) to reach consensus, regardless of the severity of failures (stopping, omission, 
or Byzantine). In [DLS88], the problem was studied using a model of partial synchrony in 
which upper bounds on message delivery time and/or processes’ relative step rates exist, 
but they are unknown a priori to the processes. The algorithms of [DLS88] are concerned 
with fault tolerance rather than timing efficiency, and therefore translate to relatively slow 


algorithms for our model. 


For our semi-synchronous model, a lower bound of (f +1)d is implied by the synchronous 
lower bound of f + 1 rounds, via a straightforward transformation of any algorithm for our 
model to an an algorithm for the synchronous model. For stopping and omission failures, 
any synchronous round-based algorithm may be simulated directly, yielding an algorithm 
for our model with a running time approximately C’ times the synchronous running time. 
This simulation strategy is described in Section 3.1. Thus, upper bounds of approximately 
(f + 1)Cd are easily derived. For Byzantine failures, it is not clear how to simulate a 


synchronous algorithm correctly. 


In [ADLS90], Attiya, Dwork, Lynch, and Stockmeyer prove nearly tight upper and lower 
bounds on the time to reach consensus in the presence of stopping failures. Surprisingly, 
they give a clever algorithm for consensus that runs in time 2fd + Cd, much faster than 
a direct simulation when C' is large. They also show a lower bound of (f — 1)d+ Cd ina 
proof that combines the arguments of the synchronous lower bound with techniques from 


asynchronous lower bounds and retiming techniques for our semi-synchronous model. 


1.2 Related work 


Current research also concentrating on the real time complexity of the consensus problem 
appears in [SDC90]. There, processes are assumed to have clocks that are synchronized to 
within a fixed additive error. In contrast to our results, the results of [SDC90] are stated 
in terms of process clock time, not absolute time. The relationship between those results 
and ours is unclear; a better understanding of the differences between two different models 


is posed as a direction for further research in Section 6.2. 


A related model is studied in [HK89] to explore the time complexity of detecting failures 
along a network path. This model assumes synchronous processes but differentiates between 
the (known) a priori worst-case bound on message delay, A, and the (unknown) actual worst- 
case message delay in a given execution, 6. Since 6 may be much less than A, it is desirable 
for algorithms to have minimal dependency on A. This model raises a concern similar to 


that raised by our model: detecting the absence of a message may be much more costly than 


receiving the message. Our algorithms run equally well in this model; we remark on how our 


bounds translate to this model in Section 6.1. 


Other work in this area includes the extensive literature on clock synchronization algo- 
rithms (see [SWL86] for a survey). Other problems recently studied in our model of timing 
uncertainty include the problem of mutual exclusion ([AL89]) and the complexity of a net- 


work synchronizer algorithm ([AM90]). 


1.3. Results of this thesis 


1.3.1 Consensus in the presence of omission failures 


In Chapter 3, we strengthen the algorithm of [ADLS90] to tolerate omission failures. The 
resulting algorithm has a running time of 4(f+1)d+Cd for n > 2f+1. This is approximately 
within a constant factor (4) of the lower bounds of (f + 1)d and (f — 1)d + Cd ({ADLS90]) 
and minimizes the dependence on the timing uncertainty C’. 


For n < 2f, a more involved analysis bounds the running time by two different quantities 


simultaneously: one bound is dependent on the ratio me, and the other is dependent on VC. 


We first derive the bound B45 + 5)(f + l)d + Cd using a finer analysis that is similar in 
spirit to the analysis for n > 2f +1. We then show that (2/C + 6)(f + 1)d + Cd is also a 


bound on the running time using a simple but different argument. 


1.3.2 Consensus in the presence of Byzantine failures 


In Chapter 4, we present a simulation algorithm using 3f + 1 processes and tolerating f 
arbitrary failures. The algorithm simulates any synchronous round-based algorithm tolerant 


of f arbitrary failures using roughly time 2C'd + d per round. 


The simulation works by keeping processes loosely synchronized to ensure that a nonfaulty 
process does not advance to round r until it has received a round r — 1 message from every 
nonfaulty process. The partial synchronization works by using a combination of two criteria 
for advancing to further phases, one based on elapsed local time and the other based on 


messages received. 


It follows that any of the known synchronous consensus algorithms tolerating f Byzantine 
failures and taking f + 1 rounds can be run in our model in time (f + 1)(2Cd + d). 
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1.3.3 Timeouts using bounded-capacity message links 


In Chapter 5, we define a realistic restriction on the message links of our model and examine 
its effect on the time needed to detect stopping failures. According to the model of [AL89] 
and [ADLS90] (used in Chapter 3), every message sent by a process is delivered within time 
d of when it is sent, regardless of the rate at which messages are sent. In reality, if a link is 
flooded with messages, their delay may be much greater. Our algorithm for omission failures 
and the algorithm of [ADLS90] ignore this consideration by requiring a process to send a 
message at every step it takes. This enables failures to be detected as quickly as possible, 
but is grossly inefficient in its use of messages. We therefore define a more realistic model of 


message delay that takes into consideration the rate at which messages are sent. 


We give a clean, modular definition of a message link of arbitrary capacity yw. Such a link 
my may be thought of as allowing the “progress” of only fs messages at any time. We then 
derive nearly tight bounds on the time needed to detect a stopping failure using such links. 
Two easy algorithms guarantee that the time between a failure and its detection is at most 
2Cd +d and C?d/p + Cd +d, respectively. We show that these bounds are nearly optimal 
by proving a lower bound of the lesser of 2Cd+d/,: and C?d/ pu +Cd4+d. 


ll 


Chapter 2 


Model and Definitions 


Our underlying formal model is essentially the same as that used in [ADLS90]. Our model 
differs by assuming for ease of presentation that all messages are delivered in the order sent 
and that processes begin executing the algorithm at the same time. The former assumption 
is not used in our algorithm for Byzantine failures and is easily removed from our algorithm 
for omission failures by employing a more complicated protocol for receiving messages. The 
latter assumption is avoided in [ADLS90] by instead providing a special individual input 
event for each process, in which it receives its initial value for the consensus protocol. In 
measuring the time complexity of the algorithm, time is measured only from the earliest 
time that all processes have received an input. Using the same formalism, our algorithm for 
omissions failures works equally well without the assumption of a synchronized start. This is 
not true, however, for our algorithm tolerating Byzantine failures, where we make use of the 
fact that all nonfaulty processes begin executing the algorithm at the same time. Without 
this assumption, the problem is complicated by the need to determine when all processes 
have received inputs. Also, in addition to allowing stronger failures than [ADLS90], we 


assume that processes know the number of failures, f, to be tolerated. 


2.1 Formal model 


We consider a system of n processes 1,...,n. Each process is a deterministic state machine 


with possibly an infinite number of states and a distinguished start state. 


A configuration is a vector C consisting of the local states of each process. Let st(z,C) 
denote the state of process 7 in configuration C’. We model a computation of the algorithm as 
a sequence of configurations alternated with events. Each event 7 is either the computation 


step of a single process or the delivery of a message to a process. The local protocol of process 
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2 consists of two transition functions, M; for message delivery events, and 5; for computation 
events. Transition function M; is applied to a state of the process and a message (taken 
from some finite message alphabet) and returns a state. (So, for example, a process can 
“remember” a message that was delivered to it.) A message delivery event 7 is of the form 
(m,2), specifying the message m delivered and the recipient process, 7. Transition function 
5; is a applied to a state of the process and returns a state and a finite set of messages to 
be sent. A computation step 7 is of the form (7, 4), specifing the process 7 taking the step 
and the set of messages M it sends in that step. (M should be interpreted as the messages 
the process actually sends at that step in the execution; if the process is faulty, this may not 


correspond to those determined by the transition function.) 


An execution is an infinite sequence of alternating configurations and events, a = Co, 71, 
Ch,...,%j,Cj,..., where Co is the vector of start states and each configuration C; follows 
from the previous configuration C;_, and the intervening event 7;, according to the state 
transitions of the process at which event 7; occurs. This means that if event 7; is an event 
at process x then (1) for y 4 x, st(y,Cj-1) = st(y,C;), (2) if x is a message delivery event 
specifying the delivery of message m then st(z,C;) is the result of applying M; to Cj-1 and 
m, and (3) if is a computation event, then st(7,C;) is the result of applying S; to Cj-1. 


Also, each message sent is delivered after it is sent and no unsent “messages” are delivered. 


A timed event is a pair (x,t), where x is an event and t, the “time”, is a nonnegative 
real number. A timed sequence is an infinite sequence of alternating configurations and 
timed events a = Co, (™,t1),C1,...,(;,t;),Cj,..., where the times are nondecreasing and 


unbounded. 


Fix real numbers c,, c2, and d, where 0 < cq < cg < «© and 0 <d< _o. Letting a bea 
timed sequence as above, we say that a is a timed execution if 

l. Co,m,C1,...,7;,Cj,... 18 an execution; 

2. The first step of each process is at time 0; 

3. There are infinitely many computation steps for each process; 

4. If x; and 7; are consecutive computation steps of the same process, then c, < t;—t; < 
C2; and 

5. If message m is sent to process 2 during computation event 7; then it is delivered to 


process 2 during message delivery event 7, 9 < k, such that 0 <t, —t; <d. 


In our timing analysis (but not in our algorithms or correctness proofs), we make the 


assumption that cz < d and therefore make the approximation d+ c & d. 


Tn all our algorithms, a process always sends the same message (at most one per step) to all processes, 
including itself. 
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2.1.1 Omission failures 


A process 7 suffers an omission failure in execution a if and only if there is a computation 
step 7; of process 7 in a specifying a set of messages that is a strict subset of the messages 
determined by the transition function 5; applied to st(?,Cj-1). Recall that computation 
step 7; specifies the messages actually sent by 2 during that step of execution a. Note that 
according to our definition of an execution, st(7,C;) must be the result of applying S; to 
st(i,Cj_-1), regardless of the messages specified by 7;. This implies that the process itself is 
“unaware” of its failure and, unless informed about it, continues executing as if it had not 
failed. (This kind of failure is sometimes called a send-omission failure.) If the algorithm 
requires 7 to broadcast a message to all processes, but 7 does not send a message to 2, then 


we say that “7 omits to 2” or that this broadcast is “unsuccessful”. 


2.1.2 Byzantine failures 


A process suffers a Byzantine failure if it changes its state or sends messages in a way not 
specified by the transition functions of the algorithm. No restrictions are made on its state 
transitions or what messages it sends, and so it may exhibit arbitrary behavior. Furthermore, 
the time between successive steps of a faulty process might not be in the interval [c1, ca]. 


The messages it sends, however, are delivered within time d of when they are sent. 


2.1.3 Consensus 


Finally, we define the consensus problem. We assume that each process begins with an 
initial binary value (its “input”) as part of its local state and may irreversibly “decide” on 
a value by entering a specially designated state. The problem is for the processes to agree 
on a binary value despite the failure of some processes. We say that a timed execution a is 
f-admissible if at most f processes fail in a. An algorithm solves the consensus problem for 
f failures within time T provided that for each of its f-admissible timed executions a, (1) no 
two different processes decide on different values (agreement), (2) if some nonfaulty process 
decides on v, then some process has initial value v (validity), and (3) every nonfaulty process 
decides by time T (time bound). Note that the validity condition does not imply termination; 
termination is implied by the third condition. We consider the binary version of the problem, 
where the initial values are 0 or 1. Like the algorithm of [ADLS90], our algorithm for omission 
failures can be extended to work for any value set, using the same extension given there 
([ADLS90], Section 5.4). Our algorithm for Byzantine failures is a general simulation for 
any rounds based algorithm and therefore can simulate any synchronous agreement algorithm 


for any value set. 
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Chapter 3 


Consensus in the Presence of 
Omission Failures 


In this chapter, we present a consensus algorithm tolerant of send-omission failures. The 
algorithm uses the same strategy as that of [ADLS90]; we first elucidate this strategy by 
describing a synchronous consensus algorithm upon which it is based and explaining our 
algorithm in terms of that synchronous algorithm. For n > 2f + 1, the running time of our 
algorithm is 4(f + 1)d+ Cd, which is approximately within a factor of 4 of the lower bounds 
of (t — 1)d + Cd and (t + 1)d ([ADLS90]). For n < 2f, the running time is bounded by two 
quantities, (3-5 +5)(f + 1)d + Cd and (2VC + 6)(f + l)d + Cd. 


In order to motivate the work presented here, we first discuss bounds attainable by more 


straightforward algorithms. 


3.1 Straightforward upper bounds 


Attiya, Dwork, Lynch, and Stockmeyer ({ADLS90]) give two simple algorithms tolerant of 
stopping failures and with running times of roughly fCd. One algorithm is based on a 
method for simulating any synchronous round-based algorithm; the other is specific to the 
consensus problem and requires that the processes begin synchronized. Both algorithms can 
be modified to tolerate omission failures without seriously affecting the running times. We 


briefly explain these two simple algorithms with the modifications. 


The first simple algorithm simulates any synchronous round-based algorithm and takes at 
most time C'd+d per round. The algorithm works by executing the round-based algorithm 
in parallel with a timeout task. The timeout task is similar to the one described at the 


beginning of Chapter 5: each process keeps a count of the number of steps it has taken and 
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at each step broadcasts the number of its current step to all other processes in the form 
“Tm alive: s” at step number s. Each process also keeps track of the “I’m alive” messages 
received from other processes and detects failures in the expected way, by detecting gaps in 
the step numbering or by the absence of messages. (We will in fact employ this strategy in 
our algorithm.) While performing the timeout task, a process simulates each round of the 
synchronous algorithm by asynchronously executing it—a process simply waits indefinitely 
on every other process for either a message of that round or the detection of that process’s 
failure. It is not hard to see that this accurately simulates the round-based algorithm: no 
process sends a round r message before receiving a round r — 1 message from all nonfaulty 
processes; A simple inductive argument shows that by time r(C'd + d) (more accurately, 
time C(d + c2) + (d+ c2)), every process has finished simulating round r of the synchronous 
algorithm. Thus, any synchronous consensus algorithm tolerant of omission failures taking 
f +1 rounds may be directly simulated to yield an algorithm for our semi-synchronous model 


that takes time (f + 1)(Cd+ d). 


Under the assumption that processes begin executing the algorithm at the same time, a 
simpler algorithm specific to the consensus problem may be used. This simpler algorithm 
does not make use of any fault-detection mechanisms. If a process starts with initial value 
1, it broadcasts a 1 and decides 1 and halts. If a process ever receives a 1 (and has not yet 
halted), it does the same. It is easy to see that if a correct process receives a 1, then some 
correct process receives a 1 by time fd and subsequently all correct processes receive a | 
by time (f + 1)d (more accurately, (f + 1)(d + c)). Therefore a process may decide 0 if it 
has run for more than (f + 1)(d + c.)/c, steps without deciding. This takes at most time 
approximately (f + 1)Cd. 


Finally, we remark that the efficient algorithm of [ADLS90] can be modified to tolerate 
omission failures by using the timeout task for omission failures outlined above. The running 
time, however, is then roughly f?d+Cd. This bound follows from a modification of the part 
of the analysis of |ADLS90] which takes the sum over each phase r of the number of processes 
that fail during the sending of an r message. Because only stopping failures are considered 
in [ADLS90], the analysis there concludes that a process may fail during the sending of at 
most one r message and therefore the sum over all r is at most f. If failures are by omission, 
then a process may fail during the sending of many r messages, but only once for any r. 
Because there are at most f + 2 phases in any f-admissible execution, the sum over all r is 


at most (f + 1)f, resulting in a bound of approximately (f + 1)fd+ Cd. 
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3.2 Intuition: the underlying synchronous algorithm 


Our algorithm and the algorithm of [ADLS90] may be interpreted as simulations of an 
underlying synchronous algorithm. In this underlying synchronous algorithm, all processes 
begin executing in round 0. In even numbered rounds, processes may decide only on 0; in 
odd numbered rounds, processes may decide only on 1. In round 0, any process with initial 
value 0 decides 0 immediately and broadcasts a message saying “I decided in round 0”; any 
process with initial value 1 broadcasts a message saying “I didn’t decide in round 0” and 
advances to round 1. In any subsequent round r, if a process did not receive a message 
saying “I decided in round r — 1”, it may decide r mod 2, broadcasting “I decided in round 
r”: if it did receive a message saying “I decided in round r — 1”, it advances to round r + 1 


broadcasting “I didn’t decide in round r”. 


It is easy to see that if a nonfaulty process decides in round r then no process decides 
in round r + 1 and all processes then decide in round r+ 2. The algorithm is also “early- 
stopping”: any execution in which at most f processes fail takes at most f +2 rounds of 
communication. (This means that all processes decide in round f + 2 or earlier, despite the 
fact that the first round is numbered 0, since a decision in round 2 is based on messages sent 
in round 7 — 1 or earlier.) The is easily seen by observing that if an execution takes x rounds 
then a faulty process decides in each of rounds 0 through x — 3: if no faulty process decides 
in round 2 < « —3 then either (1) a nonfaulty process decides in round 7 and all processes 
decide by round 7 + 2, or (2) no process decides in round 7 and therefore they all decide in 
round 7+ 1 (because no process receives an “I decided in round 2” message). Thus, f failures 
cause the maximum number of rounds, f + 2, in the following execution. All processes 
except some process jo begin with initial value 1 and advance to round 1. Process jo, with 
initial value 0, broadcasts “I decided in round 0” to all processes except some other process 
ji. Thus all processes except 7; advance to round 2; 7, decides in round | and broadcasts 
“T decided in round 1” to all processes except some process jz. This continues until finally 
process js; decides in round f—1 and broadcasts “I decided in round f—1” to all processes 
except nonfaulty process 7», which decides in round f + 1; all processes subsequently decide 


in round f +2. 
Both our algorithm and that of [ADLS90] “simulate” this synchronous algorithm, making 


several important optimizations in order to improve the running time for our model. If during 
the simulation of round r, a process receives a message saying “I decided in round r — 1”, 
it immediately advances to round r+ 1 (without waiting for round r — 1 messages from 
other processes), broadcasting to all processes, in effect, “I know of a process that decided in 
round r — 1”. Other processes in round r that receive this message relay it to all processes 
and also advance immediately to round r+ 1. A process may decide in round r only if it 


can be sure that no nonfaulty process decided in round r—1. This is ascertained only when, 
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for every other process p, either (1) the message “I didn’t decide in round r — 1” is received 
from p, or (2) p has been detected as faulty (by the timeout protocol), or (3) for some 


m 


r’ <r—1, the message “I decided in round r’” has been received from p (also remembered 


by the timeout protocol). 
The key to the improved efficiency of our algorithm relative to that of [ADLS90] is the 


addition of a mechanism for a process to detect its own failure. We require that a process 
receive at least n — f acknowledgments for every message of the synchronous algorithm that 
it sends. Until a process has received a sufficient number of acknowledgments for its round 
r message, it is prohibited from deciding in round r + 1 or advancing to round r+ 2. This 
is important to the efficiency of the algorithm because it limits to 1 the number of times a 
faulty process can omit a message of the synchronous algorithm to all nonfaulty processes. 
For n > 2f +1, the convention of waiting for acknowledgments ensures that a faulty process 
does not advance to round r+ 1 if it omits to all nonfaulty processes a message saying “I 
know of a process that decided in phase r”. If it does send such a message to a nonfaulty 
process, that nonfaulty process in turn relays it to all other processes; the faulty process 
therefore has not delayed the algorithm by very much (time d at most). The convention 
of waiting for acknowledgments requires that a process continue executing the algorithm, 


sending acknowledgments, after it has decided. 


3.3. The algorithm 


We first explain the presentation of our algorithm. We describe our algorithm as the parallel 
composition of a fault-detection protocol and a main algorithm. At each step, a process 
first executes the code of the fault-detection protocol, then executes the code of the main 
algorithm, and finally sends a message. (Recall that in our model a process may send at 
most one message at each step). 

This message is the concatenation of possibly several component “messages” which are 
specified by the queue commands in the code: if during a step, the statement “queue ‘m’” 
is executed in the code, then “message” m is a component of the message sent at the end 
of that step. We will refer to a message by any one of its components: we will say “an m 


message” or simply “an m” to refer to any message with m as one of its components. 


Our model also specifies that a process receives messages only during delivery events (and 
therefore only between process steps). For every delivery event, a process changes its state 
by adding the received message to a buffer (an unordered set). At its next step, the process 
reads and empties this buffer. A conditional statement in the code referring to the receipt 


of a message checks whether such a message was read from this buffer during the given step. 
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For ease of presentation, some components of a process’s state are not explicitly named 
or maintained in the code—for instance, the number of steps a process has taken, whether it 
has decided, or whether it has sent a certain message. Process index subscripts are omitted 


in the code but used in the text (e.g., “D;”) to refer to a local variable (D) of process 2. 


3.3.1 The fault-detection protocol 


In order to tolerate omission failures, our algorithm employs the timeout protocol described 
in Section 3.1. A process sends a message at every step that it takes, consecutively numbering 
all messages that it sends with the number s of its current step. Before a process decides, 
the message that it sends at every step is of the form “I’m alive: s”, where s is the number 
of its current step; after a process decides, the message is of the form “I’ve decided: s”. 
The failure of a process can thus be detected by a gap in the sequence numbering (recall 
we assume that message links deliver messages in the order sent) or by the absence of any 


messages for too long a period of time (more than time d + c2). 


All processes detected as faulty are added to a local set F'. When a process 7 detects the 
failure of another process 7, it broadcasts this fact in the form of a “shutdown 7” message. 
Upon receiving this message, other processes add 7 to their respective sets /’; when process 
7 receives this message, it halts, ceasing its execution of the algorithm. The timeout protocol 
also keeps track of which processes have decided. When a process receives a message “I’ve 
decided: s” from another process, it adds that process to its set D. When a process 7 adds 
j to D; (F;, resp.), it is said to have “detected” that 7 has decided (failed, resp.). We say 
that a process 2 is shut down at time t if it receives a “shutdown 2” message at time t. The 


code for the fault-detection protocol is in Figure 3.1. 


We now verify two basic properties of the fault-detection protocol with respect to arbi- 


trary executions. The first bounds the time by which a failure is detected. 


Lemma 3.1 [f at time t, process 7 omits a message to process i, andi is not shut down by 


time t+ C(d+e.)+(d+ce@) et+Cd+4d, theni adds 3 to F; by that time. 


Proof: Let s; be the step number of j at which it omits a message to 2. The lemma is 


clearly true if 7 sends a message to 2 at a step numbered greater than s; and that message 


arrives at 7 by time t+ C(d+c.)+(d+c2). If 7 does not send such a message, then 7 receives 
no message from j between time t+d and t+d+ce(1+(d+e2)/a) =t+(d+e)+C(d+ce), 
in which time ? takes more than (d + c)/c, steps and, since it is not yet shut down, adds 
to F.. | 


'As a consequence of the bound on running time to be derived, these sequence numbers are bounded by 
a function of f, d, c, and co. 
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STEP s: If “shutdown 2” received, then halt. 
If decided, then queue “I’ve decided: s” 
else queue “I’m alive: s”. 
For each j g DU F, 
if “shutdown j” message received 
then F — FU {7}; queue “shutdown j” 
if “I’m decided: s;” message received from j 
then D— DU {j} 
if “I’m alive” messages from j not numbered consecutively 
then F — FU {7}; queue “shutdown j” 
if no message received from j and more than 
(d + c2)/c, steps taken since last message received from J, 
then F — FU {7}; queue “shutdown 7”. 


Figure 3.1: The fault-detection protocol for 2 at step number s. 


The second property verifies that nonfaulty processes are never declared faulty. 


Lemma 3.2 If process1 does not fail in an execution, then i is not added to any set F; and 


is never shut down. 


Proof: For contradiction, let 7 be the process that first adds 2 to its failed set F’;. Process 7 
adds 2 to F; because either it receives a “shutdown 2” message, or it receives two “I’m alive” 
messages from 2 with a gap in sequence numbering, or it does not receive an “I’m alive” 
message from ? for more than (d+ c2)/c1 steps. 

By our choice of 7, process 7 cannot receive a “shutdown 2” message before adding 7 to 
F;—that would imply that some other process added 2 its failed set before j did. 

Because ? is nonfaulty (and the links are FIFO), 7 does not receive two “I’m alive” 
messages with a gap in the sequence numbering. 

Before it decides, 2 sends “I’m alive” messages at every step it takes and so any two 
messages are delivered to 7 at most time d+ apart (if one message is delivered immediately 
and the following message is delayed by d). In time d+ cg, 7 can take at most (d+ c.)/c1 
steps and therefore does not add iz to F;. After 2 decides, it broadcasts an “I’m decided” 
message, which causes j to add 2 to D; and prevents 7 from adding 7 to F; thereafter. Thus, 
jy cannot add z to F;. = 
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3.3.2. The main algorithm 


The main algorithm is basically an asynchronous version of the synchronous algorithm of 
Section 3.2. The code for the main algorithm appears in Figure 3.2. We call the simulation 
of round r of the synchronous algorithm “phase r”. Each process 7 starts in phase 0 with v; 
set to its own private value (1 or 0). In its first step, a process either decides 0 or advances to 
phase 1. As with the synchronous algorithm, in even numbered phases a process can decide 


only 0, and in odd numbered phases a process can decide only 1. 


When a process advances from phase r to phase r+1, it broadcasts an “r” message. (This 
is the equivalent of the message “I didn’t decide in round r” in the synchronous algorithm). 
When a process decides in phase r, it broadcasts an “r+ 1” message. (The message “r + 1” 
replaces the messages “I decided in round r” and “I didn’t decide in round r + 1” of the 
synchronous algorithm; both have the meaning “I know of a process that decided in round 
r”, so it is unnecessary to distinguish between them.) Set M” contains those processes from 
which an r message has been received. A process may decide in phase r only if it has (1) not 
yet received an r message, and therefore does not know of a process that decided in round 
r, and (2) has received an r — 1 message from all processes not yet detected as faulty or 
decided, indicating that they did not decide in round r — 1. If process 7 is nonfaulty, then 
the receipt of an r+ 1 message from 7 prevents other processes from deciding in phase r + 1 
since they do not add z to D or F before receiving it. A process that decides in round r does 
not send an r message unless it receives one first (this implies that some process decided in 
round r — | but failed). 


Our convention of acknowledging messages works as follows. Each process maintains a 
set A’ containing those processes from which a properly sequenced “ack(-,1r)” message has 


” messages is achieved by 


been received. (The restriction to properly sequenced “ack(-, 1) 
not adding a process to A’ if that process is already in F’. This restriction is necessary only 
for the bound when n < 2f.) Until a process decides, it sends exactly one acknowledgment 
message, “ack(j,r’)”, for each r’ message where r’ is less than its current phase number. After 
a process decides in some phase r, it continues to acknowledge r’ messages for r’ < r+ 1. 
This is implemented in the code by allowing the process to advance to phase r + 2 but no 
further. It is not necessary for a process to acknowledge r’ messages for r’ > r + 1 because 
as we will see, if it is nonfaulty then other nonfaulty processes do not advance to phase r+3 
without deciding and therefore do not require acknowledgments for their r + 2 messages. 
Until a process has received at least n — f properly sequenced acknowledgments for its r 


message (|A’~'| > n — f), it may not advance to phase r + | or decide in phase r. 


Definition 1 A process? is blocked in phase r (for r > 0) if it advances to phase r without 
deciding and never has |A?~'| >n— f. 
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Being blocked is a permanent state, but even if a process is not blocked in phase r, it may 
be temporarily delayed from advancing to phase r+1 as it waits for acknowledgments before 


proceeding. 


PHASE 0: If v = 1, then queue “0” and goto PHASE 1. 
If v = 0, then queue “1” and decide 0 and goto PHASE 2. 


PHASE r > 0: For each j and each r’, 1<jy<nand0<r'<r, 
if “r’” message received from 7, 
then M” — M” U {j} 
if “ack(?,r — 1)” received from j and 7 ¢ F, 
then A’! — A’ U {7} 
if 7 €¢ M™ andr’ <r and “ack(j,r’)” not yet sent, 
then queue “ack(j,1r’)”. (whether decided or not) 
If decided and M*~? 4 9 and “r — 2” not yet sent, 
then queue “r — 2” 
If not decided and |A’~'| >n— f, (enough ack’s received) 
then if MM’ 49, (some process decided in phase r — 1) 
then queue “r” and goto PHASE r + 1 
if M" = 0 and j € M""' for all j € (DUF), 
then queue “r+ 1” and decide r mod 2 and goto PHASE r+ 2 


Figure 3.2: The main algorithm of process z, performed at every step. Initially, 
a process is in phase 0 with M/" = A” = 9 for all r. 


We prove here a few basic lemmas about the main algorithm with respect to any f- 
admissible execution. The first two lemmas affirm two expected properties that held for the 


synchronous algorithm. 


Lemma 3.3 /[f some nonfaulty process decides in phase r > 0 then no process decides in 
phaser +1. 


Proof: Let 7 be a nonfaulty process that decides in phase r and consider any other process 7. 
According to the code of the main algorithm, 7 cannot decide in phase r+1 without receiving 
an r message from 2 or adding 2 to Ff; or D;. We claim that neither can happen before 7 
receives an r + 1 message from 7, which according to the code (which requires M’t' 4 O) 


precludes j from deciding phase r+ 1. First, because 2 is nonfaulty, by Lemma 3.2 it is never 
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added to F;. Process 1 may send an r message, but only after sending an r + 1 message. 
Because 7 is nonfaulty, it does not omit this message, and because messages are delivered in 
the order sent, 7 does not receive it before receiving the r+ 1 message. Process 2 is added to 
D,; only when 7 receives the message “I’ve decided” from 7. By the same argument, j does 


not receive “I’ve decided” from 7 before receiving the r + 1 message. = 


The following definition is useful in proving correctness and analyzing time complexity. 


Definition 2 Phase r is quiet if there is some process that never receives any r messages. 


Lemma 3.4 /f a nonfaulty process decides in phase r > 0 then phase r + 2 is qutet. 


Proof: By Lemma 3.3, no process decides in phase r+ 1. If a process does not decide in 
phase r + 1, then it does not send an r + 2 message until it receives one. Therefore, no 


process sends an r + 2 message and in fact no process receives an r message. a 


The next two lemmas affirm that the convention of acknowledging r messages works as 
expected—nonfaulty processes are never blocked—and the last lemma states that the failure 


of blocked processes is eventually detected by all processes. 


Lemma 3.5 For any process 1 and any nonfaulty process 7, if 1 advances to phase r > 1 
without deciding and sends anr’ message j for0 <r’ <r—1, theni receives an “ack(i,r—1)” 


message from j. 


Proof: By induction on r. Clearly the lemma is true for r = 1: jy advances to phase 1 
during its first step and sends “ack(7,0)” during the next step at which it has received a 0 
message from 2. 

Assume the lemma is true for r—1 > 1. First observe that 7 does not decide in any 
phase r’ < r—3: by Lemma 3.3, this would imply that no process decides in phase r’+ 1 and 
therefore no process sends an r’ + 2 message, but this is not possible because 7 advances to 
phase r > r’+3 without deciding and therefore must receive an r’+2 message. If 7 decides in 
phase r’ and r’ = r—2 or r—1, then j immediately advances to phase r’+2 > r after deciding 
and sends “ack(?,r—1)” to 72. Suppose that j does not decide in any phase r’ < k—1. Process 
jy must advance from each phase r’ < k — 1 because it is never shut down, has Mr # 0, and 
has |An| >n-— f: 7 is never shut down by Lemma 3.2; 7 has Mr # () because it receives 
an r’ message from 72; 7 has |A™"| > n-—f because it is nonfaulty and therefore sends an 
r” message to all processes for each r” < r’ — 1 and by the induction hypothesis receives 
“ack(j,r”)” from all nonfaulty processes—none of which, by Lemma 3.2 are ever added to 


F;. Process j therefore advances to phase r and may then send “ack(?,r — 1)” to 2. = 
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Corollary 3.6 /f process i is nonfaulty and advances to phase r > 1 without deciding, then 
it eventually has |A'~'| >n—f. (A nonfaulty process is never blocked.) 


Proof: Because 2 is nonfaulty and advances to phase r without deciding, for 0 <r’ < r it 
sends an r’ message to all processes as it advances to phase r’+ 1. By Lemma 3.5, 7 receives 
“ack(?,r — 1)” from each nonfaulty process. Because by Lemma 3.2, nonfaulty processes are 


never added to F;, each nonfaulty process is added to A’~', giving the necessary bound. 


The following lemma relies on the fact that a process continues to take steps, executing 
the algorithm after it decides; in particular, it continues to detect the failure of processes 


and, if necessary, send acknowledgments. 


Lemma 3.7 [f a faulty process 7 unsuccessfully broadcasts an r message at time t and is 
subsequently blocked in phase r +1, then all processes not shut down by time t+ C(d+c)+ 
2(d +c) = t+Cd-+ 2d detect the failure of j by that time. 


Proof: By the definition of being blocked, 7 advances to phase r + 1 but never has |A%| > 
n— f. Thus there is some nonfaulty process 7 never added to A’. By Lemma 3.5, 7 omits 
an r’ message to 7 for some 0 < r’ <r. This omission occurs at or before time t. By 
Lemma 3.1, 7 detects this failure by time t + C(d + ce.) + (d+ c2), broadcasting “shutdown 


j” to all processes in the same step. By time d+ c later, all processes not yet shut down 


have received this message and taken a step, adding 7 to their failed sets. = 


3.4 Correctness proof 


We now prove that in all f-admissible executions, the algorithm terminates and correctly 
satisfies the agreement and validity conditions. We first prove “progress”—that processes 
in fact advance to successive phases as expected. Given this progress lemma and a few 
simple facts about quiet phases, the proofs of agreement, validity, and termination are easily 
derivable. These proofs follow the same reasoning as the informal argument about the 


synchronous algorithm outlined in Section 3.2. 


Lemma 3.8 For each r > 0 and each process i that is neither blocked nor shut down in any 


phase r’ <r, process 1 either decides in some phase r’ <r or advances to phase r +1. 


Proof: For contradiction, let phase r be the first phase for which the lemma is not satisfied 
and let 2 be any process for which the lemma is not satisfied at phase r. By the choice of r, 


2 advances to phase r. 
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First note that r 4 0, since every process either decides or advances to phase | during 
its first step. 

We show below that for r > 0 and for every process j, either 7 either receives an r — | 
message from 7 or adds 7 to F; or D;. We thus derive a contradiction by concluding that 2 
may either decide or advance to phase r + 1, since it has 7 € M’~! for all 7 ¢ (D; U F;) and 
by assumption is not shut down and eventually has |A’~'| > n— f (is not blocked in phase 
r). 

Let 7 be any other process. First consider the case that 7 also is neither shut down nor 
blocked in any phase r’ < r and that further, 7 does not fail directly to 7. By the choice of 
r,j either advances to phase r or decides in a previous phase. If 7 advances to phase r, then 
it must send an r — 1 message to 7 (successfully, in this case). It cannot be that process 7 
decides in phase r — 1, since that would imply sending an r message to 7, thus enabling 2 
to advance immediately to phase r + 1, contradicting our original assumption. If 7 decides 
before phase r — 1, then it sends an “I’ve decided” message to 2 and is added to D;. 

Now consider the case that j is either shut down or blocked in some phase r’ < r or 7 
fails directly to 2. If 7 is blocked, then by Lemma 3.7, 2 will eventually detect that 7 is faulty. 
Similarly, if 7 is shut down, then it halts and z will detect its failure by timeout. Lastly, 
Lemma 3.1 ensures that if 7 fails directly to 2 and 7 is not shut down, then 7 eventually 
detects j as faulty and adds it to F;. = 


Corollary 3.9 For any r > 0, every nonfaulty process either decides in phase r’ < r or 


advances to phaser +1. 


Proof: By Lemmas 3.2 and 3.6, a nonfaulty process is never shut down or blocked; the 


corollary then follows immediately from Lemma 3.8. = 


Corollary 3.10 /f phase r > 0 is quiet, then each nonfaulty process decides in some phase 
rec, 


Proof: By Corollary 3.9, each nonfaulty process either decides in phase r’ < r or advances 
to phase r+ 1. But a nonfaulty process cannot advance to phase r+ 1: to do so, it would 


send an r message to all processes, contradicting the assumption that phase r is quiet. m 


Lemma 3.11 (Agreement) No two nonfaulty processes decide on different values. 


Proof: Let r be the first phase in which some nonfaulty process 7 decides. By Lemma 3.3, 
no process decides in phase r + 1. Because no process decides in phase r + 1, no process 
sends an r + 2 message and thus phase r + 2 is quiet. Thus by Lemma 3.10, all nonfaulty 
processes decide in some phase r’ < r+ 2. By the choice of r, all nonfaulty processes decide 


in either phase r or phase r + 2, in either case on r mod 2. = 
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Lemma 3.12 (Validity) If any process decides on value b, then some process i starts with 
Vvi= b. 


Proof: Clearly if some process j decides on 1, it does so in phase r > 0 and that process 
itself must have started with v; = 1 since otherwise it would have decided on 0 during its 
first step. 

If some process j decides on 0, it cannot be that all processes started with v; = 1. 
For then, no process would decide in phase 0 and no process would send a 1 message. No 
process would receive a | message and therefore no process would advance to phase 2 without 
deciding and so no process would decide 0. = 


Lemma 3.13 (Termination) In any f-admissible execution, there is a quiet phase num- 
bered at most f +2 and so each nonfaulty process decides in some phase r < f +2. 


Proof: If some nonfaulty process decides in phase r < f then no process decides in phase 
r+1 and no process sends an r+2 message. Phase r+2 is therefore quiet and by Lemma 3.10 
all nonfaulty processes decide by phase r +2 < f +2. 

If no nonfaulty process decides in any phase r < f, then there must be a phase h, 
0 <h < f, in which no faulty process decides, and therefore in which no process decides. 
If a process does not decide in phase h, then it does not send an h + 1 message until it 
receives one. Therefore no process sends an h + 1 message—phase h + 1 is quiet—and by 
Lemma 3.10, all nonfaulty processes decide by phase h +1 < f +41. = 
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3.5 Analysis of time bounds 


We now bound the amount of real time until all nonfaulty processes decide in any f- 
admissible execution. The analysis in this section is carried out with respect to any given 
f-admissible execution. Having already proved the correctness of the algorithm, we will here- 
after assume d >> cg and make approximations appropriately. We first establish the tools for 
our analysis and then conclude with the nearly optimal bound for n > 2f +1 (Section 3.5.1) 


and two bounds for n < 2f (Section 3.5.2). We first introduce some notation. 


e For r > 0, let t, be the earliest time by which all processes not blocked in any phase 
r’ <r of the execution have either decided, advanced to phase r + 1, or been shut 
down. 


Because every process either decides or advances to phase | on its first step, to = 0. 
e Let phase h be the first (smallest numbered) phase that is quiet. 


e For r > 0, let B, = {7:7 is blocked in phase r + 1}; let 6, = |B,|. 


The definition of B, may seem unusual, but makes sense on closer analysis. We will want 
to bound t, — t,_1, which we think of as the time for phase r, in terms of the number of 
processes that omit an r message to all nonfaulty processes. This number is 0,, since all such 


processes are subsequently blocked in phase r + 1. 


Lemma 3.14 Forr 41’, B.O By =O. 


Proof: By definition, a process must advance to phase r’ in order to be blocked in phase 
r’. Ifr <r’ andz € B,, then 2 is blocked in phase r+ 1 <r’ and cannot advance to phase 
r+2<r'+1 or greater. Therefore, 7 is not blocked in phase r’ +1 and cannot be in B,.. m 


Corollary 3.15 S“!=> b; < f. 


Proof: By Corollary 3.6, a nonfaulty process is not in any B,, so these sets consist of faulty 
processes only. The bound of f then follows immediately from the disjointness of the sets 
B,, from Lemma 3.14. = 


We prove our upper bound by summing the times of the individual phases. We will say 
“the time of/for phase r” to mean t,—t,_1;. We prove an upper bound for two kinds of phases: 
those that are quiet and those that are not. We first derive some useful lemmas about the 
receipt of acknowledgments. We then prove an upper bound on the time to complete any 


phase—in particular, quiet phases. We then prove a lemma (Lemma 3.19) that is at the 
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heart of the timing analysis, regarding causal chains of r messages. The time for phases that 
are not quiet depends on whether or not n > 2f + 1 and will be deferred until the following 
subsections (Sections 3.5.1 and 3.5.2), where we will also sum over the phases to derive the 


total time bounds. 


We first prove a useful lemma about the timeliness of acknowledgments: if a process 
receives a sufficient number of properly sequenced acknowledgments for its r message, then 


it receives them promptly, by time t,_1 + 2d. 


Lemma 3.16 For r > 0, if process j eventually has |A’| >n— f, then it has |A"| > n — f 
by time t, + 2d. 


Proof: Process 7 sends an r message either as it advances to phase r + 1 or as it decides in 
phase r — 1. If process 7 broadcasts its r message because it advances to phase r + 1, then 
it is clearly not blocked in any phase r’ < r and is neither decided nor shut down before 
it broadcasts this message, and so broadcasts it by time t,. Similarly, if 7 broadcasts its 
r message because it decides in phase r — 1, then it does so by time t,_1. In either case, 
jy broadcasts its r message by time ¢t, and any process that receives an r message from 7 
receives it by time ¢, + d. 

Consider any process 7 € A’. We claim that ¢ sends “ack(j,r)” by time t, + d. By the 
fact that it sends “ack(j,r)” eventually, process 2 must advance to phase r + 1 or greater 
(either by deciding in phase r—1 or phase r or by advancing to phase r+1 without deciding) 
before sending “ack(j,r)”. It follows that ¢ is neither blocked in any phase r’ <r nor shut 
down before it does so and therefore advances to phase r+ 1 by time t,. By time t, + d, 2 


also receives an r message from 7 and therefore sends “ack(j,r)” by then. = 


Corollary 3.17 For r > 0, if processi sends anr +1 message after time t, or for some 7 
process? sends “ack(j,r)”, then 2 has |A’| > n—f by time t, + 2d. 


Proof: If process 7 sends an r+ 1 message after time t, then it does not send the r + 1 
message as a result of deciding in phase r, since processes that decide in phase r do so by 
time t,. Therefore 7 sends an r+ 1 message as a result of advancing from phase r+ 1, which 
requires that it have |A™| > n— f. By Lemma 3.16, 7 therefore has |A%| > n — f by time 
tp + 2d. | 


We now prove a generous upper bound on the time to complete any phase (in particular, 


quiet phases). The proof is very similar to the proof of progress. 
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Lemma 3.18 ¢; — to < Cd4+d and for any phase r > 1, 
t, <max(t,-1 + Cd +d, t,-2 + Cd + 2d). 


Proof: For contradiction, assume that for r > 1 (respectively, r = 1) at time max(t,-1 + 
Cd+d, tp. + Cd + 2d) (resp. time to + Cd + d), some process 7 has neither decided nor 
advanced to phase r + 1 nor been shut down and is never blocked in any phase r’ < r. By 
this time, by the definition of ¢,_1, 7 is in phase r, and since it is not blocked in phase r, has 
|A'~"| >n—f by Lemma 3.16. We will reach a contradiction by showing that 7 must decide 
in phase r by this time because for every other process j, either 2 receives an r — 1 message 
from j or 7 detects that j has decided or failed (7 € D; U F;). 

Let 7 be any other process. First consider the case that 7 (1) is not blocked in any phase 
r’ <r —1, (2) is not shut down by time t,_1, and (3) does not fail directly to 7 before or at 
time t,_1. By Lemma 3.8, 7 either advances to phase r or decides in some phase r’ < r — 1; 
by definition it does so by time t,_1. If 7 advances to phase r, then it sends (successfully, by 
assumption) an r — 1 message to 7 by time t,_1 and 2 receives this message by time t,_1 + d. 
If 7 decides in phase r’ < r — 1, then by time ¢,, + d < t,_1 +d, 7 receives an “I’ve decided” 
message from 7 and adds 7 to Dj. 

Now consider the case that 7 either (1) is blocked in some phase r’ < r — 1 (2) is shut 
down by time t,_1, or (3) fails directly to 2 at or before time ¢,_1. If 7 is shut down or 
fails directly to 2 at or before time t,_,, then by Lemma 3.7, 7 detects the failure by time 
t,-1 + Cd-+d. Case (1) is not possible for r = 1, so we are finished for that case. If 7 is 
blocked in some phase r’ < r — 1, then because it advances to phase r’, 7 neither decides 
nor is blocked nor shut down in any prior phase. Therefore, by time ¢,_1 < t,_2, 7 advances 
to phase r’, broadcasting (unsuccessfully) an r’ — 1 message. By Lemma 3.7, all processes, 
including 2, detect the failure of 7 by time t,_2 + Cd + 2d. = 


In bounding the time of a phase r that is not quiet, we will bound the time until every 
process receives an r message (which every process does, by the definition of a quiet phase). 
By that time, every process that is not yet decided or shut down or blocked in any phase 
r’ <r may advance to phase r + 1; thus this is a bound for ¢,. In bounding the time until 
every process receives an r message, the following reasoning is at the heart of the analysis. 
In order for the first r message to ever be sent, some process must decide in phase r — 1, 
which by definition, it does by time t,_;. An r message sent by any other process 7 that does 
not decide in phase r — | is sent because 2 received an r message. Thus, a causal chain of 
r messages may be followed and the first r message received by any process can be traced 
back to a process that originated it (2; in the following lemma), sending the “first” r message 
before t,_;. Because a process broadcasts an r message as soon as it receives one (at its 
next step, to be precise; also, assuming it has |A’~'| > n — f, which it does after ¢,_1 + 2d 


if at all), our time bound for phases that are not quiet is approximately d times the length 
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of the shortest such chain to each process. We now prove a lemma about the existence of 
such chains and their basic timing properties. This lemma is central to every bound we will 


prove for the omission failures algorithm. 


Lemma 3.19 I/f phase r is not quiet, then for every process 19, there exists a sequence of 
distinct processes 19,11,...,%, and messages Mo,m1,...,Mz with k > 0 such that 

(1) for0 <j <k, a; sends the first r message, mj-1, received by t;-1, 

(2) exactly one process, ip, sends an r message by time t,_1, and 

(3) for0 <j <k, process i; sends an r message (m,;_1) by time tp4 + (k —j + 1)d. 


Proof: Phase r is not quiet, so every process 7; receives an r message; let m; be the first 
r message that 7; receives. Define a sequence of processes 2g, 21,... inductively as follows: if 
2; sends an r message by time f,_1 then define k = 7 and let 21; be the last process of the 
sequence; otherwise, define 2;41 to be the process that sends m,. 

We first claim that the resulting does not include repetitions and is therefore finite. This 
is clear if 79 sends an r message by time t,_1 (then & = j = 0). If not, we show that for 
any 0 <j <k, process 7; is distinct from processes %9,...,2j-1. Only to may fail to send 
an r message. If it does, then clearly it is distinct from the other processes in the sequence; 
if not, then let m_, be any r message that it sends. If 7; sends an r message by time ¢,_1, 
then clearly it is distinct and we are done. If not, then for all 7,, 0 <x < j, because 2, 
sends an r message (m,_1) later than time t,_1, 7, must send it as the result of receiving 
an r message (by the definition of t,1, a process that decides in phase r — 1 broadcasts r 
by time t,_1). It follows that the sending of m,-1 by ?, is preceded by the sending of mz, 
the first r message received by 2,. Because a process broadcasts an r message only once, it 
follows that processes %o,...,2; are distinct. 

Thus the sequence z,,...,21,%9 forms a chain of processes such that for 0 < 7 < k, process 
2; sends the first r message, mj_1, received by 2;_; and k is the only process in the sequence 
to broadcast an r message before time t,_1. This proves (1) and (2). 

It remains to show (3), the timing property. For 0 < j < k, the fact that 2; sends an r 
message but does not decide in phase r — | implies that 2; advances to phase r by time ¢,_1, 
since it is not blocked in any phase r’ < r and is not decided or shut down before sending 
m,—1, Which it does after time ¢,_;. Therefore, by time ¢,_1 + 2d, each 2; is in phase r and 
by Lemma 3.16, has |Ar"| >n— f. Since 27z_1 receives mz_1 by time t,_1 + d, it advances 
to phase r + 1, sending mz_2 by time t,_1 + 2d. Process 7,_2 receives this message by time 
t,-1 + 3d and thus advances to phase r + 1, sending mz_3 by time t,_, + 3d. Similarly, for 
0< yj <-k, process 2; receives m; and sends mj;_1 by time t,-1 + (1 +k —j)d. rT 


To complete the lemmas necessary to tightly bound the running time, we need only bound 


the time for any phase that is not quiet. This bound depends on whether or not n > 2f +1. 
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3.5.1 Bound for n> 2f+1 


We show that the algorithm depends on C' only to the extent of an additive factor of Cd. 
For C large, this algorithm may be far more efficient that a direct rounds simulation. The 


bound we obtain for n > 2f +1 is within approximately a factor of 4 of optimal: our bound 


is 4(f + l)d + Cd; the lower bound proved in [ADLS90] is (f — 1)d + Cd. 


Having bounded the time for quiet phases in Lemma 3.18, we need only bound the time 
for any phase that is not quiet. If n > 2f +1, we can be sure that when a faulty process 
broadcasts an r message, it ezther sends to at least one nonfaulty process or becomes blocked 
in phaser+1 since f < n—f. Ifit sends to a nonfaulty process, then that process will send an 
r message to all processes and the phase will end. The number of processes blocked in phase 
r +1 is exactly 6,; our bound for phase r is roughly 6,-d. This is the key difference between 
our algorithm and the algorithm of [ADLS90]: a faulty process may cause delay d only if 
it sends exclusively to other faulty processes; the convention of requiring acknowledgments 


ensures that each faulty process can do so only once. 


To reinforce the intuition about this bound, we first describe how this bound is realized 
by a worst-case execution: Process 1 € B, is the first to send an r message. It decides in 
phase r — 1 at time t,_1 (no later, by definition of t,-1, since process 1 is not blocked in 
any phase r’ <r — 1) and sends an r message to only process 2 € B,. Process 2 waits until 
time ¢,_, + 2d for |AS7'| > n — f and then, having received an r message from 1, advances 


to phase r + 1, sending an r message to only process 3 € B,. The pattern is repeated until 


process 6, + 1 ¢ B, receives an r message at time t,_1+(6,+1)d. Process b, + 1 advances to 


phase r+1 and omits an r message to exactly one nonfaulty process, 7. All nonfaulty process 


except 7 receive an r message from 6, + 1 at time t,_1 + (b, + 2)d and 7 receives an r message 
from them at time t,_1 + (b, + 3)d. By this time, each process has either advanced to phase 
r+1 (as it sent an r message), decided, been shut down, or is blocked in some phase r’ < r. 
This scenario shows where the extra 3d arises: one d is caused by the delay of waiting (by 
process 2 in this scenario) for acknowledgments from the previous phase, another d is for a 
faulty process (here, b, + 1) that is not blocked in phase r + 1 to send an r message to a 
nonfaulty process, and another d is for the remaining nonfaulty processes (here, 7) to receive 
an r message. (In [ADLS90], only the last extra d is incurred; this leads to the factor of 2 


in their bound, instead of 4 in ours.) 


Lemma 3.20 For n > 2f +1 andr > 1, tf for all r’ < r phase r’ is not quiet, then 
t, —tpy < (34+ 6,)d. 


Proof: We show that by time ¢,_1 + (3+ 8,)d, all processes receive an r message. Thus, by 
that time, every process is either decided, shut down, blocked in some phase r’ < r, or may 


advance to phase r+ 1. 
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By Lemma 3.19, we know that for every process zo, there is a sequence of distinct pro- 
cesses, 29,21,-..,2% satisfying the three properties of Lemma 3.19. 

Now, if k < 6, +2 then tg receives mo by time t,_1+(k-—1+1)d4+d < t,-1+(b,+2)d+4d. 
If k > 6, +2, then there is a j such that k-— 6, <7 <k andz; ¢ B,. By Lemma 3.19, 7; 
sends an r message by time t, + (k —j + 1l)d <t,-1+ (14+ 6,)d. Because i; ¢ B,, 7; sends 


an r message to at least n — f > f + 1 processes, one of which must be nonfaulty. This 


nonfaulty process, ¢, receives an r message from 7; by time t,_1 + (2 + },)d. 

We now conclude the proof by showing that process ¢ sends an r message, received by 
all processes, by time t,_1 + (2 + 6,)d. Because no phase r’ < r is quiet, it follows from 
Lemma 3.4 that ¢ does not decide in any phase r’ < r — 2. If € decides in phase r — 1, then 
does so, sending an r message, by time t,_1. If € decides in phase r or advances to phase r 
without deciding, then it does so by time t,_; and subsequently sends an r message once it 
receives one and has |Ai~'| > n—f, which, by Lemma 3.16, it does by time t,_1+(2+6,)d. m 


We can now bound tightly the running time of any f-admissible execution by summing 


the bounds for all phases in that execution. 


Theorem 3.21 For n > 2f +41, the algorithm above solves the consensus problem for f 
omission failures within time 4(f + l)d+ Cd. 


Proof: For any given execution, let A be the first quiet phase. By Lemma 3.10, each 
nonfaulty process decides in some phase r < h, by time ¢,. If h = 0 then by Lemma 3.10, 
each nonfaulty process decides in phase 0 in its first step and the running time is 0. [fh = 1 
then by Lemma 3.10, each nonfaulty process decides in phase 1 or 0, and by time t;; by 
Lemma 3.18 the running time is ty — tp < Cd-+d. 

If h > 1, then we can bound the time for phases 1,...,2 — 1 by Lemma 3.20, and the 
time for phase h by Lemma 3.18. Thus we have 


h-1 
tn—to = So(tp — bya) + (tn — tra) 
fat 
< S°(34+6,)d+ (Cd + d) (by Lemmas 3.20 and 3.18) 
r=1 
< (f+1)8d+ f-d+(Cd+d) (by Lemma 3.13 and Cor. 3.15) 
= 4(f+1)d+Cd. 


For C > 4, it is possible to construct an execution that takes exactly time 3d + 4(f — 
3)d+3d+Cd+d. In this execution, the first phase takes time 3d, the following f —3 phases 
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take time 4d, the penultimate phase takes 3d and and the last phase takes time Cd + d. 
Each of the phases taking 4d develops when all processes receive an r — 1 message at time 
t,-1 and all but one, p,_1, advances to phase r. Process p,_; decides on r — 1 mod 2 at t,_1 
(before it receives the r — 1 messsage) and sends an r message to exactly one other process, 
Pr41, which receives its acknowledgments for its r— 1 message at time t,_; + 2d and sends an 
r message to exactly n — f processes. By time t,_, + 4d, all processes receive an r message 
and, except for one process, p,, advance to phase r +1. In the following phase, at time 
t, + 4d, process p,41 decides (the processes to which p,41 omitted an r message run slowly 
and do not detect its failure until t,-1 + 2d + (Cd +d) = t,-1 + 7d = t, + 3d, so it is not 


shut down before then). Remaining details are left to the reader. 


3.5.2 Bounds for n < 2f 


When n < 2f, we are able to bound the running time of the algorithm in two ways, yielding 
one expression that depends on the ratio i and another expression that depends on the 
square root of C’. We will use Lemmas 3.14 (B, 9 B,. = ), 3.16 (the timeliness of acknowl- 
edgments), 3.18 (the time for any phase), and 3.19 (sequences of causal r messages), and 
Corollaries 3.15 (the sum of the 6,), and 3.17 (also regarding acknowledgments) the proofs 


of which did not rely on the relative values of n and f. 


Bound dependent on i 


This bound requires a lemma about the length of causal sequences of r messages more 
complicated than Lemma 3.19. Processes not in B, must send an r message to n — f 
processes but not necessarily to a nonfaulty process. We therefore are not able argue as 
for n > 2f +1 that phase r ends very soon after a process not in B, sends an r message. 
Nevertheless, disregarding processes in B, for the moment, if it were true that a process 
could not get an acknowledgment from another process that already sent an r message, then 
it would take at most time (<b)d before a nonfaulty process received an r message. Our 
algorithm does not exactly enforce this restriction on acknowledgments, but it does prevent 
a process from using acknowledgments received from a process that previously omitted an r 
message to it. We are thus able to derive a bound of B25 +b, +4)d below in Lemma 3.23. 
This argument is most easily made by considering a directed graph on the faulty processors. 


Accordingly, for a given execution, define 


e directed graph Gi = (VJ, Ef) where 
VJ = {all processes that fail during the given execution}. 


Ef = {(t,j):i sends an r message to j; 7,7 € VJ; 74 J}. 
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e 6/(7,7) = length of the shortest path in G/ from 7 to 7, where 7,7 € VJ. 
e Sir. = {7:7 sends an r message by time t,_1}. 


e S"f — {7:7 sends an r message to a nonfaulty process}. 


Claim 3.22 If phase r is not quiet and no nonfaulty process decides in phase r — 1, then 
there exist faulty processes a € Str. and y € S™! such that there is a path in G! from a 
to ¥. 


Proof: Let + be the first process to send an r message to a nonfaulty process. Process 7+ 
must be faulty: by the choice of y, no process sends an r message to a nonfaulty process 
earlier than y sends an r message and therefore if y is nonfaulty then no process sends 
an r message to y before it sends its r message. Therefore y must decide in phase r — 1, 
contradicting our assumption that no nonfaulty process decides in phase r—1. We conclude 
that y € S"/. Note that 7 sends an r message before any nonfaulty process does. 

Let C; be the nodes in G/ from which node ¥ is reachable (including y) and let a be 
the process such that no process in Cy sends an r message before a does. It follows that a 
sends an r message no later than y does. Because no nonfaulty process sends an r message 
before 7 does, a does not receive an r message from a nonfaulty process before sending its 
r message. By choice, a does not receive an r message from any faulty process before it 
sends its r message. We therefore conclude that a receives no r messages before sending its 
own, and therefore must decide in phase r — 1, sending an r message by time t,_1 (by its 
definition). rT] 


Lemma 3.23 Forr > 1, if for all r’ <r phase r’ is not quiet, then 
ty — tra < (324 + by + 4)d. 


Proof: We show that by time t,_; + B45 +b, + 4)d, every process receives an r message. 
Thus, by that time, every process that is never blocked in any phase r’ < r and is neither 
decided nor shut down at that time, has |A"~'| > n — f by Lemma 3.16 (because it is not 
blocked in phase r) and therefore may advance to phase r + 1. 

First note that if a nonfaulty process decides in phase r — 1, it does so by time ¢,—1, 
broadcasting an r message that is subsequently received by all processes by time t,_1 4+ d, 
and the lemma is proved. So we consider the case that no nonfaulty process decides in 
phase r— 1. 

Lemma 3.22 applies for this case: it says that there exist processes 7 € S'-1 9] VJ and 
3 € S™! OVS such that j is reachable from 7 in Gf. Let a € St-1 and y € S™ be a closest 
pair of nodes in GI: 

5!(a,7) = min F 62 (2,3). 


tp 
ies,’ —tavJ 
jest avs 
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We first bound the time by which y broadcasts its r message. Fix a minimal length path 
from a to y and let 9 be the last process on that path that sends an r message by time 
tp + 2d. We claim that y broadcasts its r message by time t,_1 + (6/(8,y) + 2)d. By 
the choice of 3, each process 7 on the path from / to 7 sends an r message later than time 
t,-1 + 2d and therefore by Corollary 3.17 has |At"| >n—f by time t,_; + 2d. By time 


t,-1 +3d, the process on this path after 3 receives an r message (from 3) and thus broadcasts 


an r message. Similarly, for j > 1, the 7” process on the path after 3 sends an r message 
by time ¢,_1 + 2d + jd and process 7 sends an r message by time t,_1 + (2 + 6/(6,7))d. 

We now show that by time t,_1 + (6/(3,7) + 4)d, all processes receive an r message. A 
nonfaulty process 7 receives an r message from ¥ by t,-1 +(3+6/(3,y))d. Because no phase 
r’ <r is quiet, it follows from Lemma 3.4 that 7 does not decide in any phase r’ < r — 2. If 
n decides in phase r — 1, then it does so by time t,_1, sending an r message as it does; all 
processes receive it by time t,_1+d. If advances to phase r without deciding, then it does so 
by time t,_1. By Corollaries 3.6 and 3.17, 7 has |A’~"| > n—f by time t,1 + (3+6/(6,7))d. 
By this time, 7 has received an r message from y and therefore if 7 has not yet sent an r 
message—if 7 has not yet advanced from phase r or has decided in phase r and advanced to 
phase r + 2 but not yet sent an r message—it may then send an r message. An r message 
is then received by all processes by time t,_1 + (6/(3,y) + 4)d. 

To complete the proof, we now show 6/(8,7) < i + b,. Let k = 6f(8,7) and let 
L; = {p: 6!(8,p) =7} for 1 <i<k. Define Lo = {3} and L_, = 9. Consider the sum 


k-1 
C= S- |Li-a U L[; U Lisi. 


1=0 
Since the sets L; are disjoint, each node in GJ is counted at most 3 times, soo < 3|G/| < 3f. 


Claim 3.24 For0 <i<k-—1, at least k — 6, of the sets L;_,UD;U Lj41 has cardinality at 
least n — f. 


Proof: Clearly, for 2 < k—1, no set L; is empty, since y, at distance k from {, receives 
an r message from a faulty process. At least k — 6, sets L; contain a process j ¢ B, such 
that j is on the path from { to 7. For each j, and each process ¢ in A‘, clearly, 7 sends 
€ an r message; we will show that if 7 is on the chosen path from £ to y, then process ¢ 
sends 7 an r message also. We will also show that if 7 € L; where: < k —1, then @ is faulty 
and therefore in G/. Thus, for all ¢ in Af such that 7 € L;, there are edges of Gf in both 
directions between j and ¢ and if 7 ¢ B,, then |L;-1 UL; U Ligi| > n — f, completing the 
proof. 

We first show that if 7 € L; where: < k —1, then ¢ € A® is faulty. If € were nonfaulty, 
then j would be in S"/. But j cannot be in $”, since y, at distance k, was defined to be 
the closest node to a in S77 VS but 7 € L; is at distance i < k. 
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We next show that for each 7 on the chosen path from / to y, if  € A’, then ¢ sends 7 an 
r message. Consider first the case that @ broadcasts an r message before sending “ack(j, 1)” 
to j. Since ( € A™, € does not omit a message to j before sending “ack(j,1r)”; in particular, 
it does not omit the r message. Consider then the case that ¢ does not send an r message 
before sending “ack(j,r)” to 7. By the choice of 3, j sends its r message (and @ receives it) 
later than time t,_1 + 2d. Because @ sends an “ack(j,r)” message, ¢ either decides in phase 
r — | or advances to phase r without deciding. However, & cannot decide in phase r — 1: 
processes that decide in phase r — 1 do so, sending an r message by time t,_1, but we are 
assuming € does not send an r message before sending “ack(j,r)”, which is later than time 
t,_1 + 2d. Thus, at the time that @ receives the r message from j, ¢ is either in phase r, 
not yet having sent an r message, or decided and in phase r + 2, not yet having sent an r 
message. In its first step after receiving the r message from 7, process & queues both “r” and 


“ack(j,r)”. Because ¢ € A’, this message is not omitted, so ¢ sends an r message to j. 


Thus we have (k — b,)(n — f) < o and o < 3f, or 6/(B,y) = k < ae + b,, which 


completes the proof: all processes receive an r message by time t,_1 + (25 +b,+4)d. 


Theorem 3.25 Forn < 2f, the algorithm above solves the consensus problem for f omis- 
sion failures within time B25 + 5)(f +1)d+ Cad. 


Proof: For any given execution of the algorithm in which A is the first quiet phase, by 
Lemma 3.10, each nonfaulty process decides in some phase r < h, by time ¢;,. Again, if 
h = 0 then by Lemma 3.10, each nonfaulty process decides in phase 0 in its first step and 
the running time is 0. If h = 1 then by Lemma 3.10, each nonfaulty process decides in phase 
1 or 0, and by time ¢,; by Lemma 3.18 the running time is t, — to < Cd+d. 

If h > 1, we can bound the time for phases 1,...,4 —1 by Lemma 3.23, and the time for 
phase h by Lemma 3.18. Thus we have 


h-1 
th—to = So (te — tra) + (th — tha) 
ht 
< X(4 | 3 + b,)d + (Cd + d) (by Lemmas 3.23 and 3.18) 
< (f+1)(4 35d f-d+(Cd+d) (by Lemma 3.13 and Cor. 3.15) 
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Bound dependent on JVC 


The analysis of the previous section shows that the running time of our algorithm is 
(54+ 3-5)(f+1)d+ Cd. 


Note that if f is close to n, say n = f +1, then the bound is roughly proportional to f7d, 
which is no improvement on the algorithm of [ADLS90]. However, for these proportional 
values of n and f, we are better able to bound the running time. The analysis of this section 


shows that the running time of our algorithm is also bounded by 


(2VC + 6)(f + 1)d+ Cad. 


So when 2VC +6 < 5+ Bch, or roughly n < f + f/VC, we have a better bound on the 


running time. 
Define a partition the first r phases of the given execution into two classes according to 
their length: 
X, = {p:t,—tp1<VC-d and p <r} = {short phases} 
Y,. {p:p¢X, and p <r} = {long phases}. 


and define 


Sf = {i: 7 omits an r message to a nonfaulty process after time t,_1}. 
We can bound the short phases by their defined bound, but bound the long phases by 


chains of r messages, via the following two lemmas. 


Lemma 3.26 [f phase r > 1 is not quiet then either t, < t,-1 + (|SJ|+3)d or 


all nonfaulty processes decide by this time. 


Proof: We once again show that by time t,_1+(|S/|+3)d, all processes receive an r message 
and thus by this time are either decided, shut down, blocked in some phase r’ < r, or may 
advance to phase r+ 1. 

By Lemma 3.19, we know that for any process zo, there is a sequence of distinct processes 
20,21,---,2% such that k is the only process in the sequence to broadcast an r message before 
time t,_1, and for 0 <j <k, by time t,_1 + (1+ —7)d, 2; sends the first r message, mj-1, 
received by 2;-1. 

Now, if k < |S/] then zo receives mo by time t,_4+(k—-1+1)d+d <t,_,+(|S/|+1)d. If 
k > |S/|, then there is a 7 such that 0 < k—|Sf| <j <k andi; ¢ SJ. Process 2; therefore 
sends an r message to all nonfaulty processes by time t,_4+(1+k—j)d <tp4+(14+|S/])d. 
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If some nonfaulty process has not yet decided, then it sends an r message to all other 
processes by time ¢,_1 + (2+ |S/|)d and process 29 receives an r message by time t,4 + (3 + 


IS). . 


The key observation is that a process cannot fail to a nonfaulty process in many long 
phases: 


Lemma 3.27 For any execution of the protocol taking at least ¢ phases and for any process 
j, there are at most /C +3 phases p € Yu such that 7 € St, 


Proof: If 7 omits an k message to a nonfaulty process at time ¢t then by Lemma 3.1, that 
nonfaulty process detects 7’s failure by time t+ Cd +d, broadcasting “shutdown )” at that 
time. We have t < t;, and so 7 is shut down by time t+ Cd+2d <t, + (VC + 2)/Cad. It 
follows that there are at most /C +2 long phases € such that t, < te < Cd+ 2d. Thus 3 
cannot attempt to send (and cannot omit) an ¢+ 1 message after time t, and is therefore 
not in Si for any p> £. = 


Theorem 3.28 Forn < 2f, the algorithm above solves the consensus problem for f omis- 


sion failures within time (2/C + 6)(f + l)d+ Cad. 


Proof: Let phase fh be the first quiet phase. Again, if h = 0 then by Lemma 3.10, each 
nonfaulty process decides in phase 0 in its first step and the running time is 0. If h = 1 then 
by Lemma 3.10, each nonfaulty process decides in phase 1 or 0, by time t;; by Lemma 3.18 
the running time is t; — to < Cd+d. 

If kh > 1, we consider two cases. Consider first the case that not all nonfaulty processes 
decide in phase h — 2. We bound the length of the short phases by their defined length. 
We bound the length of the long phases by Lemma 3.26 and then sum the sizes of Si using 
Lemma 3.27. The length of phase h is bounded by Lemma 3.18. Thus we have 


th—to = DD (to —toalt DO (tp tyr) + (th — tha) 
pEXp-1 pEYn-1 
< |X|: VCd+d SS (3 |S/]) + (Cd+d) (by Lemmas 3.26 and 3.18) 
pEYn-1 
< |Xp-1|- JVCd + 3|Yn-1|d + f(VE + 3)d + (Cd +d) (by Lemma 3.27) 
< (f+ 1)VCd43(f +1)d4+ f(VC4+3)d+Cd4d (by Lemma 3.13) 


A 


(2VC + 6)(f + Id + Cd. 
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Now consider that case that all nonfaulty processes decide in phase h — 2. The running is 
then bounded by tp_2 — to: 


th-2 — to 


IA 


» (tp —tp-a) + » (to — tp-1) 


pEXp—2 


pEYn—2 


|Xn-2]- VCd + SO (3+ [SZ] )d 


pEYn—2 


(by Lemma 3.26) 


|Xp—2]- VCd + 3|¥,-2|d + f(VC +3)d (by Lemma 3.27) 


fVCd + 3fd + 
(2VC +6) fd 


L f/Cd + 3fd 
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(by Lemma 3.13) 


Chapter 4 


Consensus in the Presence of 
Byzantine Failures 


In this chapter we present a simulation algorithm using 3f + 1 processes and tolerating f 
Byzantine failures. The algorithm simulates any synchronous round-based algorithm tolerant 
of f Byzantine failures and uses time r(d + 2C'd) + Cd, where r is the number of rounds 
required by the synchronous algorithm. 


The simulation works by keeping processes loosely synchronized to ensure that a nonfaulty 
process does not advance to round r until it has received a round r — 1 message from every 
nonfaulty process. The partial synchronization works by using a combination of two criteria 
for advancing to further phases, one based on elapsed local time and the other based on 
messages received. A similar technique is used in [WL88] to initiate new rounds of clock 
resynchronization. In particular, our criteria for ending round 1 is essentially the same as 
the criteria used in [WL88] for ending every round; our criteria for subsequent rounds is 
different. 


4.1 The simulation algorithm 


The algorithm simulates a synchronous algorithm by ensuring that each nonfaulty process 
receives all round r messages of the synchronous algorithm from all other nonfaulty processes 
before advancing to round r+ 1. We do not explore here the formal semantics of “a correct 
simulation”; rather we regard as sufficient the following correspondence ensured by the above 
property: For every execution of the simulation, there is an execution of the round-based 


synchronous algorithm in which the nonfaulty processes receive the same vector of messages 
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from each other at each round. Since the behavior of faulty processes is not restricted, clearly 


the corresponding synchronous execution is legal. 


Therefore, for the purposes of simulation, we define a synchronous algorithm by its mes- 
sage function only, suppressing information about the state of the synchronous algorithm. 
Let M;(r,V"~') denote the vector of messages to be sent in the synchronous algorithm by 
process 7 in round r when the ordered set of messages V"~! is received in round r — 1 (of 
course, this message function may also depend on the state of the process; we leave this 
implicit). Without loss of generality, assume each process sends a message to all processes 


at every round of the synchronous algorithm. 


Recall that we assume all processes begin executing the algorithm at the same time. At 
each step, a process increments a counter s (initially 0) and executes the code in Figure 4.1, 
explained below. A local variable, initially 1, keeps track of the ROUND number. Ordered 
set V" contains the r‘” message received from each process. We refer to the r‘” message sent 
by a process as a “round r” message. (Recall that we assume each process sends a message 


to all processes in every round of the synchronous algorithm.) 


Each process first sends its round 1 message and then waits for at least time d to ensure 
that it receives a round | message from every other nonfaulty process. When it can be sure 
that time d has elapsed, it advances to round 2 and broadcasts its round 2 message based 
on the round | messages it has received so far. It ensures that time d has passed by either 
waiting for d/c, of its own steps or by receiving f + 1 round 2 messages—this ensures that 


some nonfaulty process has waited at least time d. 


In subsequent each round r, a process waits for at least time 2d (actually 2d + 3c.) after 
at least f + 1 nonfaulty processes have sent a round r message. By this time, all nonfaulty 
processes must have received at least f + 1 round r messages and therefore advanced to 
round r and sent a round r message. At this time, a process advances to round r + | and 
broadcasts its round r+ 1 message. Again, there are two ways for a process to deduce 
that sufficient time has passed: if it takes (2d + 3c2)/c1 steps after receiving at least 2f + 1 
round r messages or if it receives at least f + 1 round r+ 1 messages. The latter ensures 
that some nonfaulty process has advanced to round r +1 and therefore has already waited 
a sufficient amount of time (at least time 2d after at least f + 1 nonfaulty processes sent a 


round r message). 


4.2 Correctness 


Let t, be the latest time that any nonfaulty process sends a round r message. Again, we 


assume that all processes begin at the same time (here, t,). We say a process “advances 


4] 


ROUND | Send M(1,-); goto RouND I’. 
Round I’ Ifs > d/eq or |V?| > f +1, 
then goto ROUND 2 


ROUND r Send M(r,V"—'); goto ROUND r’. 
ROUND r’ If |V"| > 2f +1, 
then s — 0; goto ROUND r”. 
ROUND r” If s > (2d4 3e2)/aq or |V"T"| > f +1, 
then goto ROUND r+1. 


Figure 4.1: The simulation of a synchronous algorithm. At each step, a process 
increments the counter s and executes the code according to Its 
present round number. V” is the ordered set consisting of the 
rt message received from each process. M(r,V’—') denotes the 


message function of the synchronous algorithm for round r. 


to round r” when it executes the “goto ROUND r” statement in the code. In order to 
prove correctness, we must show that a nonfaulty process eventually advances to all rounds 
required by the synchronous algorithm and always receives a round r message from all 


nonfaulty processes before advancing to round r + 1. 


Lemma 4.1 Each nonfaulty process advances to all rounds required by the synchronous al- 
gorithm. 


Proof: By induction on the round number. Clearly each nonfaulty advances to round 2—it 
advances to round 1’ after its first step and advances to round 2 after at most 1+ d/c; more 
steps. 

For r > 2, assume all nonfaulty processes have advanced to round r. Then all nonfaulty 
processes have sent a round r message and advanced to round r’. Since n — f > 2f +1, 
there are at least 2+ 1 nonfaulty processes, so each nonfaulty process eventually receives at 
least 2 + 1 round r messages and advances to round r”. After at most (2d + 3c2)/c1 steps 


in round r”, each nonfaulty process advances to round r + 1. = 


Lemma 4.2 No nonfaulty process advances to round r+1 before receiving a round r message 


from each nonfaulty process. 
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Proof: By induction on the round number r. 

r = 1: Each nonfaulty process takes more than d/c, steps before advancing to round 2. 
Thus each advances later than time ¢; + d, which is after the round 1 message of each 
nonfaulty process, sent at time ¢,, is delivered. 

r > 1: Assume the lemma is true for r — | (i.e., no nonfaulty process advances to round 
r before receiving a round r — 1 message from each nonfaulty process). We show the lemma 
is true for r. Let 7 be the first correct process to advance to round r+ 1 and let 7; be the 
time at which ¢ advances to round r” (by Lemma 4.1, this time is well-defined). We make 


the following series of deductions about the events that occur at or before the listed times: 


7; : Because 7 is in round r”, by the induction hypothesis, 7 has received 
a round r — 1 message from all nonfaulty processes. Because 7 has 


advanced to r”, it has received at least 2 + 1 round r messages. 


7 +d: All nonfaulty processes are in round (r — 1)’ or greater (because they 
have each sent an r — 1 message to 7) and have received at least 2f + 1 


round r — 1 messages (from each other). 
7 +d+c,: All nonfaulty processes therefore advance to round (r — 1)”. 


7 +d+2c,: All nonfaulty processes have received at least f + 1 round r messages 
(from the nonfaulty subset of processes that sent round r messages to 7) 


and therefore advance to round r. 


7 + d+ 3c, : Allnonfaulty processes send a round r message and advance to round r’. 


7; + 2d + 3c, : All processes receive a round r message from each nonfaulty process. 


Because by choice 7 is the first nonfaulty process to advance to round r + 1, it follows 
that 2 advances to round r+ 1 only after (2d + 3c.)/c, steps in round r”, which occurs later 
than time 7; + 2d + 3c). We conclude that 7 receives a round r message from each nonfaulty 
process before advancing to round r+1. Since all nonfaulty processes advance to round r+1 


after 7, they also receive a round r message from all nonfaulty processes before advancing. m 


4.3. Analysis of time bounds 


Again, we assume d >> cg and therefore approximate d+ c) by d in the timing analysis. 
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Lemma 4.3 t,—1t, < Cd and, forr > 2, tp, —t, <d+2Cd. 


Proof: Clearly, tg < ¢t, + Cd. By time t, all nonfaulty processes send a round r message 
and advance to round r’. Therefore by time ¢, + d, all nonfaulty processes receive at least 
2f +1 round r messages and advance to round r”. Within another time 2Cd, all nonfaulty 
processes have taken (2d + 3c.)/c, steps and advanced to round r + 1, sending an r + 1 


message. a 


Theorem 4.4 There is an algorithm using 3f + 1 processes which solves the consensus 


problem for f Byzantine failures within time Cd+ f(d+ 2Cd) = fd+ (2f + 1)Cd. 


Proof: Any (f+1)-round synchronous algorithm can be simulated. Agreement and validity 
follow from correct simulation. Termination follows from Lemma 4.1. The time bound 


follows from Lemma 4.3. | 
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Chapter 5 


Bounded-capacity Message Links and 
Failure Detection 


In fault-tolerant distributed algorithms, a common primitive for detecting failures is to “time 
out” failed processors. If processors fail by simply stopping, then a failure may be detected 
by the absence of messages from a processor. In this chapter, we consider how quickly such 


failures can be detected in our semi-synchronous model. 


If it is assumed that all messages sent are delivered within time d of when they are sent, 
then the following simple protocol minimizes the time between any failure and its detection. 
(This is the strategy employed in the algorithm of [ADLS90] and our algorithm of Chapter 3.) 
Each processor broadcasts a message at every step that it takes. If no message is received 
from another processor for more than (d + c2)/c1 local steps, that processor is declared 
faulty. Because local steps are separated by at least time c,, at least time d+ cy passes 
before this many steps are taken. Because local steps are separated by at most time c2, the 
time between the delivery of any two consecutive messages sent by a processor is at most 
d+cg. It follows that only failed processors are declared faulty. The maximum time between 
any failure and its detection is approximately C'd + d, occurring in the following scenario: 
processor p broadcasts a message at time ¢ and then fails; these messages are delivered at 
time t + d; every other processor runs slowly (its steps separated by c2) after t+ d, and thus 
p’s failure is not detected until time t + d+ @(d+c@)/q xt+d4+Cad. 


Although the above protocol guarantees minimal delay between any failure and its detec- 
tion, it is clearly inefficient in its use of messages. It takes advantage of the strong assumption 
that all messages are delivered within time d, regardless of the rate at which they are sent. 
In reality, the performance of a message link may suffer if messages are sent too frequently. 
In this chapter, we propose a model of message links with bounded capacity and analyze the 


effect of the capacity bound on the efficiency of detecting stopping failures. 
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5.1 Modeling bounded-capacity links 


Define a message link of unit capacity and delay d as a communication channel that queues 
incoming messages in FIFO order and delivers the first message in the queue within time 
d of the later of when the message is sent and when the previous message is delivered. 
(For simplicity, we will assume that message links deliver messages in the order sent. Our 
algorithms do not make use this assumption and our lower bounds hold in spite of it.) For 
positive integer yu, define a message link of capacity and delay d as the composition of ps 
message links each of unit capacity and delay d/j, connected serially so that messages are 
delivered from link z to link z+ 1 for 1 <7 < yp and link p delivers messages to the recipient 
process. Messages are neither lost by a link nor delivered out of order, and once a processor 


has sent a message, it cannot cancel that message. 


Thus, in the absence of any other message traffic, the delay of a single message is bounded 
by w-d/u = d. Note that if a single component link delays all messages by the maximum 
amount, d/j1, then messages are delivered at a maximum rate of js messages per time d. In 
particular, it is easy to see that if the last component link delays each message by d/j, then 


for any interval of time of length /, at most lanl messages are delivered. 


On the other hand, if no two messages are sent within time d/j: of each other, then each 
message is delivered within time d of when it is sent. This is easily seen by an induction on 
the number of messages sent. Assume the previous message is delivered by the 7“ sublink 
within time 7-d/ of when it is sent (clearly this is true for the “first” message ever sent). 
If message m is sent at time ¢, then for 0 <7? < p, by time t +7-d/j the previous message 
has been delivered by link 7 + 1 and m is delivered by link 2 (by induction on 7). Thus m is 


delivered to the recipient process within time yu - d/j = d of when it is sent. 


For the lower bound, we assume only that in the worst case, a link delivers every pair of 


messages at least time d/j: apart. 


5.2 Timing out failed processors 


We will consider a system of processors fully connected by message links of capacity yu and 
delay d. These processors may fail by stopping. A process is said to detect the failure of 
another processor when it irrevocably decides that the other has failed. A timeout protocol 
is correct if it satisfies two properties for all executions and all processors p and q: (1) if p 
fails and ¢g does not fail, then g eventually detects the failure of p, and (2) if neither p nor ¢ 
fails, then neither p nor q detects the failure of the other. 
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For a given execution a, we say that p detects the failure of g within time T in a if q 
fails at time t in a and p detects the failure of g at time t’ <t+T in a,. We say a timeout 
protocol guarantees a detection time of T if for all processors p and gq and all executions a 
in which p fails but ¢g does not, ¢ detects the failure of p within time 7’ in a. 


Because in our model each pair of processors is connected by a private bidirectional 
message link, we will assume that the timeout protocol executes independently for each 
pair of processors. We will therefore prove bounds on detection time for a system of two 


processors, p and q. 


5.2.1 Upper bounds on detection time 


An upper bound of 2C’d+d is achieved by a simple protocol that works for any link capacity. 
The two processors continually exchange a single token message: when p receives the token 
message from q, it sends a token message back to q, and g does likewise. If a processor 
takes more than (2d + c.)/c, steps without receiving a message, it concludes that the other 
processor is faulty. Because there is at most one message in transit at any time, it is always 
delivered within time d of when it is sent. Clearly a nonfaulty processor is never timed out. 
This protocol guarantees that any failure is detected within time 2C'd + d (to be precise, 
d+ C(2d+4 c2) + c; recall we approximate d+ c = d): if p fails at time t, then by time t+ d 


all of the messages it has sent are delivered to g and q has sent its last message to p; within 
another time c2(1 + (2d + c2)/a1) & 2Cd, ¢ has taken enough steps to conclude that p has 
failed. 


An upper bound of C?d/pu + Cd +d is achieved by a protocol in which each processor 
sends a message every (d/j)/c, steps. A process concludes that the other has failed if it 
has taken more than (Cd/j: + d)/c, steps without receiving a message. Clearly, the sending 
times of every two messages are separated by at least time d/j and therefore, as shown 
in Section 5.1, each message is delivered within time d of when it is sent. The maximum 
amount of time between the delivery of two consecutive messages from a given processor is 
then ¢o(d/s)/c1 +d = Cd/u-d (if the first message is delivered immediately and the following 
message incurs the maximum possible delay, d). This is less than the minimum amount of 
time, Cd/u +d+ «aq, that the other processor waits before detecting failure. This protocol 
guarantees a detection time of C*d/u+Cdt+ d: if p fails at time t, then by time t+d all of the 
messages it has sent are delivered to g; within another time c2(Cd/p+d)/e, = C?d/u4+ Cd, 
q has taken enough steps to conclude that p has failed. 


Thus we obtain a simple upper bound of min(2Cd + d, C?d/u + Cd +d). Note that 
29Cdtid«< C7d/p + Cd+d if and only if uw <C. 


AT 


5.2.2 Lower bounds on detection time 


We now prove a nearly corresponding lower bound of min(2C'd + d/pu, C?d/u + Cd +d). 
Note that 2Cd + d/pu < C*d/u + Cd +d if and only if ¢ < C+1. Thus, the bounds are 
tight except for uw < C+1: when C < p< C4+1, C?d/p + Cd-+4d is the best upper bound 
and 2C'd + d/, is the best lower bound; when p < C, 2C'd +d is the best upper bound and 
2C'd+d/, is the best lower bound. 


We first prove that there exists some execution in which p runs “fast” (its steps separated 
by time c,), gruns “slowly” (its steps separated by time c2), messages from q to p are delivered 
immediately, messages from p to q are delayed by at least time d, and some pair of consecutive 
messages from p to q are separated by at least time d/j:. We prove that such an execution 
exists for any protocol guaranteed to detect failures within any bounded amount of time. 
This is proved below using the properties of the bounded-capacity message links. The idea 
is that if the last component link from p to q delays all messages by d/j: then the delivery of 
every pair of messages is separated by time d/j. Therefore, if each pair of messages sent by 
p were separated by less than d/j, then messages would be sent (put onto the link) faster 
than they were delivered (removed from the link). Thus the number of messages sent but 
undelivered and, consequently, the total delay of a message, would grow in time without 
bound. The time between when p fails and when gq receives no further messages is therefore 
increased without bound. 


Lemma 5.1 For all B and for any correct timeout protocol that guarantees a detection time 
of B, there exists an execution in which 
1. All consecutive steps of p are separated by cy; 
All consecutive steps of q are separated by co; 
All messages from q to p are delayed by time 0; 
All messages from p to q are delayed by at least time d; 


For all to, there exists a pair of messages m, and mz sent by p at times ty and ty 
respectively, such that ty >to, tg —t, > d/p, and no message is sent by p in the 
interval (ty, ta). 


Proof: For contradiction, suppose not. Fix any execution 3 of the protocol in which (7) the 
first three timing constraints are satisfied, (¢7) each component link from p to q¢ delays each 
message by time d/j1, and (222) no processor fails. Such an execution exists because conditions 
(z), (22) and (222) are independent of each other and within the bounds of the model. Clearly, 
condition (22) implies that the fourth timing constraint is satisfied—all messages from p to 
q are delayed at least time d. We prove that the fifth condition is also satisfied in 3. To do 
so, assume for contradiction that it is not. 
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First note that because is infinite, p must send an infinite number of messages: if it 
does not, then let my be the last message that it sends and consider an execution y in which 
p fails after sending my. Because q receives the same messages from p in each execution, 
it cannot distinguish between the two executions and therefore g either does not detect p’s 
failure in y or erroneously decides that p has failed in /. 

Recall that a processor can send messages only during steps and p’s steps are separated 
by exactly time c, in 7. It follows that if two consecutive messages are not separated by 
at least time d/j, then they are separated by at most k = 2/4] — 1 steps, which is time 
kee <d/p. 

Consider the interval [to,¢9 + x] of execution 3, where x is defined below. Because p 
sends an infinite number of messages and, by assumption, every two consecutive messages 
are separated at most time k +c, process p sends at least |x/(k-c,)| messages in this 
interval. But since the last component link delays each message by d/j, at most || 


messages are delivered in this interval. Thus the number of messages sent but not delivered 


xv 


keey 
links, the last message sent in this interval may not be delivered until all prior messages have 


in this interval is at least ( 


—l)- (Hh +1). According to the properties of the message 


been delivered. Thus the last message sent by p in this interval may not be delivered until 
time top ta+ "ton — in 2): Let x be any number large enough so that ‘(2 - —2)>B 
(recall that k- a < d/p). 

We conclude that the last message sent by p in the interval [to, t9 + x] of 3 is not delivered 
until after time tp + « + B. Since p does not fail in 3, ¢ does not time out p; in particular, ¢ 
does not time out p before time tp +2+B. However, before time tg ++ B, this execution is 
indistinguishable to g from an execution in which p fails at time t9 +2 and which is otherwise 
identical to @ at p and q¢ up to times tg + x and tg +: « + B, respectively. Therefore in this 
execution g does not detect the failure of p within time B. This is a contradiction on the 


assumed protocol. = 


Our lower bound proof uses the retiming techniques of “shifting” events in time and 


“shrinking” portions of executions that were developed in [AL89] and [LL84]. 


Theorem 5.2 In a system with links of capacity and delay d, no correct timeout protocol 
can guarantee failures to be detected within less than time min(2C'd+d/, C?d/+Cd+d). 


Proof: Let T = min(2Cd+d/p, C*d/u+Cd+d). For contradiction, assume the existence 
of a protocol that guarantees a detection time of 7’. We do not make use of the particular 
value of T until the final step of the proof (the construction of execution 3”). We will reach 
a contradiction by showing that there is an execution of the protocol in which p does not 
fail but ¢ decides that it has. 

Let $ be an execution of the protocol whose existence is implied by Lemma 5.1 with 


to = d (25). Let m, and my be the two messages specified by the lemma, sent by p 
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d 
titd/us (m1) 
t2 
(fast) 1 (slow) 
(m2) etitd/utd 


Figure 5.1: Execution 3, the existence of which is proved by Lemma 5.1, takes 
the above form except that messages from p to g may be delayed 
more than d and messages may be sent by g at arbitrary times. The 
events of p (q) are on the left (right), with time represented by the 
vertical dimension. An arrow represents a message labelled with its 
delay, with its tail at the time of the send event and its tip at the 
time of the receive event. 


at times ¢t, and ft, respectively. Figure 5.1 depicts an example of an execution satisfying 
Lemma 5.1; for presentation, messages from p to q are shown taking exactly time d and 
messages from g to p are shown at arbitrary times. 

Let a be an execution in which (7) events at p are identical to those of 3 up to time fy, 
(72) p fails at time ty after sending mj, and (272) events at ¢ are identical to those of @ up to 
time t; + d/ +d. Clearly a exists, since messages from p to q are delayed by at least time 
din § and so q doesn’t receive mz until tg +d >t, +d/u4d. Also, the assumed protocol 
guarantees that in a, g detects the failure of p before time t, + T. 

The rest of the proof proceeds as follows. By retiming the events of a and {, we construct 
executions a’ and 3’, which are indistinguishable to both p and q from a and 3 respectively. 
By retiming the events of a’, we construct a”, which is indistinguishable from a’ to q. Finally, 
by retiming the events of 3’, we construct 3”, which is indistinguishable from (’ to p up to 
the time that it sends mz and indistinguishable from a” to q up to the time that it times 
out pin a” Thus, although p does not fail in 8”, ¢ times out p in 8”, contradicting the 


correctness of the assumed protocol. 
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d ty-—d 
d 
—(d-—d 
1 ti —(d—d/j) 
ty 0 ty 
d 
(fast) itd/u (slow) 

@t,47-d 


Figure 5.2: In the region of interest, execution a’ is simply a with events of 
processor g occurring earlier in time by d. Because p fails at time 
ti, g detects the failure of p by time ¢; + 7’ — d, denoted by the 
circle. 


Construction of executions a’ and 3’ 


Conceptually, we wish to construct a’ from a by letting each event at q occur earlier in time 
by d (“shifting” those events earlier by d). Strictly speaking, this may not be possible for 
all events at g because of initial conditions. However, it is sufficient to shift by d the events 
of g that occur after time t; in a and “shrink” some interval before t, in a (i.e., retiming 
the events of the inverval so that q runs fast in that interval of events in a’). In particular, 
we shrink the interval [0, Kd. Note that by our choice of to = sd in choosing £, the 
last event of this interval occurs before m, is sent at time t,. Leaving all events at time 0 
unchanged, steps of g in this interval, which take time c, in a, are retimed to take time cy 
in a’. Thus the interval is shrunk by a factor of C and the last event of the interval occurs 
earlier in a’ by sd — syd =d. Figure 5.2 depicts the suffix of a’, showing the shifted 
events of the region in which we shall be interested. 

This execution satisfies the timing constraints on message delivery, since messages sent 
by p (delayed by at least din a) are received by g at most d earlier in a’ and hence are 
delayed by at least 0 in a’; messages sent by q (delayed by 0 in a) are sent at most d earlier 
in a’ and hence are delayed by at most din a’. 

Execution 3’ is constructed similarly, shifting earlier by d the events at ¢ in (3. 
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Because p and g do not know the time between any particular pair of steps they take, 
they cannot distinguish between either a and a’ or 3 and (’. It follows that a’ and ’ are 
not distinguishable to p up to the point at which it fails and not distinguishable to q up 
to when it receives mz in 2" (at least time tp > t, + d/p). Also, q’s detection of p’s failure 


occurs before time #4; + T — din a’. 


P : *to—to/C q 


d—(d—d/n)+4(4—d/u) 


(slow) 


ty ty 


(fast) utd/u 


(fast) 


@ t1+4(T-d) 


Figure 5.3: Execution a” is constructed from a’ by mapping the interval [t; — 
(d—d/w), t1 +(T —d)| of a’ to the interval [#; —A(d—d/p), t+ 


+(T — d)] of a” and appropriately shifting the rest of q’s events. 


Construction of execution a” 


Recall that g runs slowly in a@ and most of a’—its steps are separated by c.. We now 
construct a” from a’ by a retiming certain events at g. Events at p are the same as in a’ up 
to time t,, when p fails in both executions; after time t,, the events (message deliveries) at 
p may be defined arbitrarily within the bounds of the model. 

The retiming operation at qg maps the interval [t; — (d — d/u), t1 + (T — d)] of a’ to 
the interval [t; — (d — d/u), t1 + 3(T — d)] of a” by letting ¢ run fast over this interval 
in a”. Events at time ¢, in a’ also occur at ¢; in a”; events in the above interval of a’ are 
retimed to occur closer to time ¢, by a factor of C. The rest of execution a’—before time 
t, — (d —d/) and after time ¢t + (7 — d)—is shifted merely to preserve the step times of 
events on the borders of this interval. To be precise, a” is defined at g for by retiming each 


event that occurs at gq at time t’ > aad in a’ to occur at g at time t” in a”, where t” is 
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defined as follows: 
+ (d—d/u)—gld—d/p) if Aydt’ <t,—(d—d/p) 
M= 4 t+ 4(t'—t1) if t,-(d—d/p) <t<t,+(T-d) 
—(T—d)+3(T -d) if #>t4(T-d) 


This execution is illustrated in Figure 5.3. 


Again, we need to shift the events before time t; — (d — d/j) while preserving 
initial conditions. To do this we partially undo the shrinking performed on the interval 
[(0, “<d] of a. These events were mapped to the interval [0, +; 
fast, with the last event of the interval occurring exactly time d earlier in a’ than in a. 


d] of a’, in which q runs 


In a”, we need the last event of this interval to occur exactly time (d—d/j)-—4(d—d/,1) 
later than in a’. Because this amount is less than d, we are able to do this, in effect 
partially undoing the original shrinking. The timing assumptions for steps of q are 
clearly satisfied. Because the net effect from both shrinking operations is to shift any 
—"d of a earlier by less than d in a”, the timing 
assumptions for message delivery are also clearly satisfied, for the reasons outlined in 


particular event in the interval [0 


the discussion of a’. 


By construction, this retiming operation does not cause violations of the bounds on 
process step times. We now verify that a” is consistent with the timing assumptions for 
message delivery. First note that all events at p before time ¢,; occur at the same time in 
executions 3,a,a’ and a”. We show that for any event at g occurring at time ¢” in a” and 
at time t in a (and hence at t’ = t —d in a’) such that t” < t, and t > sd, we have 
t—d<t" <t. By the retiming mapping above, as a function of t’ we have 


<0 <4 (d~dlu)— S(d—d/p), 


(this is because t’ is mapped forward in time furthest when t’ < t; — (d — d/j); least when 
t’ = t,) which, substituting t’ = t — d, gives 


1 
t-d <t" <t—d/p ald d/j) < t. (5.1) 


In a, every message from p to q is delayed by at least d. We claim that in a”, every 
message from p to q is delayed by at least time 0 and by less time than in a. If a message is 
delivered at g after time ft; in a”, then because p sends no messages after time ¢,, it must be 
sent by t; (no new message receipts at q have been introduced to a”) and hence delayed at 
least time 0; also, events at ¢ after time f; in a” occur earlier in a” than in a, so the message 
is delayed by less than it is in a. If a message is delivered at g at time t” < t; in a” then by 
Equation 5.1, it is delivered earlier in a” than in a by not more than d; it follows that the 


message is delayed by at least time 0 in a”. 
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We also claim that the delay of each message from q to p in a” is delayed by at least 0 
and at most d. In a, all messages from gq to p are delayed 0; if in a” they are sent before fy, 
then from Equation 5.1 they are sent earlier (and delayed more) by not more than d. Any 
message sent by qg after t, is defined arbitrarily to be within the bounds of the model. 

Finally, we note that ¢g detects the failure of p before time t, + a(T —d) ina”. 


Construction of execution (” 


We now construct execution 3” in which p does not fail and which is indistinguishable to q¢ 
from a” up to time t; +3(L'—d). In proving that 8” satisfies the timing assumptions on step 
time and message delivery, we will make use of the fact that T = min(2Cd 4+ d/p, C?d/p + 
Cd +d). Because ¢ times out p before time t; + 4(T' — d) in a”, we conclude that in 8”, ¢ 
mistakenly times out the nonfaulty p, contradicting the assumed correctness and completing 
the proof. 

To construct 8” at q we use exactly the same events as in a”, up to time ¢; + a(T —d). 
We do not specify the events occurring at qg later than this except to say that any message 


sent by p after time t, is received at g at least time d later. 


P : *to—to/C q 


(slow) 


t— a (d—d/u) 
t t 


(not fast) 


t+4(T-d)-d (fast ) 


© 414+4(T-4) 


Figure 5.4: Execution 3” is essentially the same as execution a”, except that p 
does not fail; instead, it runs slowly after sending message my, and 
message mz, Is delayed by d. Because p sends no other messages 
before my, this execution appears the same as a” to g until It 


receives mp. 
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At p, we construct 8” from 3’ by mapping the interval [¢,, t; + d/] of 3’ to the interval 
[t1, ti + 4(£ — d) — d] of 8” (p runs fast over this inteval in $’; it runs more slowly over 
this interval in 2”). Events in this interval are retimed to occur further from time t, by at 
most a factor of C’ (as we will show). We do not specify events occurring at p after time 
ty + 3(T — d) —d except to say that any message sent by q after time t; — 4(d — d/,) is 
delivered at p exactly time d later. Thus, 9” is defined at p for t” < t, + 4(T — d) —d by 


retiming each each event that occurs at time t’ in 3’ to occur at time t” in 2”, where t” is 


“ t if t < ti 
= 1 
+E) if << tu 


This execution is illustrated in Figure 5.4. 


defined as follows: 


We now verify that 8” is consistent with the timing assumptions of the model. Note that 
all events of 3” at p before time ft, are the same in 3’, 3,a,a’, and a”; events of 3” at ¢g before 
time t, + a(T — d) are by definition the same as in a”. Having already verified the timing 
properties for a”, we need verify only the timing properties involving events (processor steps, 
message sends, and message receipts) occurring at p in the interval [f1,t, + 4(¢ — d) — d]. 
Events occurring at p later than t, + a(T —d)—d and at q later than ¢; + a(T — d) are 
inconsequential to the proof and may be scheduled in any way consistent with the bounds 
of the model. 

First, we verify that successive steps of p after t, are separated by at most cg. We show 
that for any interval [¢7, t”] of 3”, mapped from the interval [#;,¢;] of 8, where ty <t; <t; < 
1 +d/p, we have t! —t/ < C(t; — t,): 


1 
Ue —yg@-d)-d 
Wat = yy 
4(C7d d)—d 
< a, —1) het Cd = 4 
d/p 
= (t;—t)C. 


It follows that because any two steps of p are separated by time c; in 3’, they are separated 
by at most C +c. = c in 9". 

We now verify that messages sent by p after ¢; are within the proper bounds. The first 
message sent by p after ty is m2, which in 9’ (and (9) is sent at tg >t, + d/p and thus in 2” 
is sent at t) > t,+ a(T —d)—d. Messages sent by p after time t, are specified to be delayed 
by at least time d, so mz is not delivered until at least time t; + 4(T —d). (Note that q 
times out p by this time.) The delivery of mz and all subsequent messages by p is consistent 
with our definition of 8” at q. 

We now verify that messages from g to p are within the proper bounds. We analyze these 


messages in three cases according to when they are sent by q in execution 3’ (which is the 
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same time as they are sent in a’). 
Case 1: q sends at time t’ < ty — d in pi’. 
A message sent by g at time t; —d in 9’ is sent and delivered at time ¢; in 3. Since the events 
at p are the same in 3” and { and a”, messages sent by g before t; — d in (” are delivered 
to p by ¢, in 8” and are therefore, by the analysis of a”, are delayed by correct amounts. 
Case 2: q sends at time t’ in 2’ where ty) —d <t' < t, — (d—d/p). 
Such a message is delivered at time t/+ d in 9’ where ty < t@ +d <t,+d/p. In 2” the 
sending event at q is mapped to time t! + (d — d/) — 4(d — d/w), which is less than ¢,. In 
B" the delivery event at p is mapped to 4, + ailal(L d) — d)(t' + d—1t,), which is greater 
than t,;. Thus, this messages is delayed by at least 0. We show below that it is delayed by 


at most d: 
ty Oe dt) = [+ (d= dfn) — Bld aly) 
= f+ (1) r+ (2 -*) (d —t,) G d/ 1) a(d a1) 
cy I = (d= dp) $d) — [ls = dal + (= au) — Rd) 
since t' <t, —(d—d/j) and (oot) >0 
< h+a(r-a) d [t a(d a1) 
< 4 ocd + d/j —d)—d4 a(d d/{) since T < 2Cd4+d/t 


C 
d. 
Case 3: q sends at time t’ > t, — (d—d/) in ’. 


These messages are sent at t” > t; — 4(d—d/w) in 8” and thus are defined to be delivered 


at p exactly time d later. Note that such messages are delivered at p later than time 


1 1 1 1 
t, - —d+—d d = t,+2d+—d/u-—-—d-d 
1 ait Gain + 1 + Gali a 
1 
1 
> h+a(T-d)—d 


This is consistent with our definition of 8” at. p. 
Thus we conclude that @” is a valid timed execution in which p does not fail but ¢ times 


out p. This is a contradiction on the correctness of the assumed protocol. = 
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5.2.3. Bounds for two processors using a single message link 


We remark that our techniques give a tight upper and lower bound of C?d/y+Cd-+d for a 


system of two processors with a message link in only one direction. 


In such a system, we have two processors, p and q, and a single message link of capacity 
ji from p to gq. Naturally, a protocol does not need to detect failures of g. All other previous 
definitions apply. 


The second simple protocol described in Section 5.2.1 operates independently in each 
direction. It immediately gives a protocol for the unidirectional case, guaranteeing that in 
any execution, g detects the failure of p within time C?d/y + Cd + d. 


It is also not difficult to see that our lower bound proof of Theorem 5.2 specializes to the 
unidirectional case to give a corresponding lower bound of C?d/j + Cd-+d. Theorem 5.2 is 
proved for 7 = min(2Cd + d/p, C?d/p + Cd+d). A similar theorem for the unidirectional 
case may be proved with T = C?d/y+Cd+4d. Recall that in that proof, the value of the 
timeout detection time T’ guaranteed by the protocol is not used before the claims about 
execution 3”. All preceding claims except those involving messages from qg to p carry over a 
fortiori. Lemma 5.1, for example, is true also for the unidirectional case with the exception 
of its third condition, which regards messages from q to p. The proof of Theorem 5.2 uses 
the fact that T < 2Cd +4 d/p in claims about 8” only to verify bounds on the delay of 
messages from g to p. This analysis is not needed for a theorem about the unidirectional 
case and hence the entire proof specializes to the unidirectional case to give a lower bound 


of C2d/ + Cd+d. 
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5.3 Consensus with bounded-capacity message links 


We remark on how our upper bounds for consensus are affected by bounding the capacity of 


the message links used. 


5.3.1 Byzantine failures 


Because it is not message-intensive, our algorithm for Byzantine failures is not affected by 
the restriction of bounded-capacity message links. Recall that the algorithm for Byzantine 
failures does not include a fault-detection task and does not require a process to send a 
message at every step it takes. The correctness follows from the fact that at least time 
2d passes between the time that the round r — 1 messages of all nonfaulty processes are 
delivered and the time that any nonfaulty process advances to round r+ 1. The round r 
message of a nonfaulty process can incur more than delay d only if it is sent before the 
previous message is delivered. The previous message is its round r — | message, so even if 
the round r message incurs added delay, it is still delivered by time 2d (actually, d) after 
the round r—1 messages of all nonfaulty processes are delivered, and all nonfaulty processes 
receive it before advancing to round r+ 1. Otherwise, if the round r message does not 
incur added delay due to the capacity of the message link, the proof of Lemma 4.2 holds as 
before. Because nonfaulty processes do not send any messages other than the messages of 
the synchronous algorithm, it is easy to see that the delay of messages does not affect the 


proof of Lemma 4.3, and the running time is not affected. 


5.3.2. Omission failures 


First note that if every pair of messages sent by a process are separated by at least time 
d/j:, then each message is delayed by at most time d and the omissions algorithm is not 
affected (the analysis of Chapter 3 holds). Because the fault-detection protocol requires a 
process to send a message at every step it takes, messages may be separated by as little as 
time c,; therefore the omissions algorithm is not affected if the message links are of capacity 
> d/c. 


The first consequence of using links of capacity « < d/c, is the obvious effect on the 
fault-detection protocol. Instead of the bound Cd+d guaranteed for the time until a failure 
is detected (shown in Lemma 3.1), a bound of only min(2C'd 4+ d/p, C?d/u + Cd +d) can 
be guaranteed by the fault-detection protocol. Lemmas 3.7 and 3.18 then also involve the 


above expression instead of C'd + d. 
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But a more serious affect on the running time of the algorithm is the added delay between 
when a process “should” send a message (according to the main algorithm) and when it may 
send it. A crucial element of the algorithm is the piggybacking of messages of fault-detection 
task and messages of the main algorithm. A straightforward implementation would require 
that messages of the main algorithm can only be sent during steps in which a message of 
the fault-detection task is to be sent. 


If the first timeout task, in which each pair of processes continually sends a “token” back 
and forth, were used for the fault-detection protocol, up to time 2d may elapse between 
when a process is required to send a message of the main algorithm and when it is able to 
piggyback that message onto a message of the fault-detection task. Thus each message may 
in effect be delayed by a total of 3d, since the timeout task ensures that all messages are 
delivered within time d of when they are sent, despite the capacity of the message links. 


This gives bounds of 


f4+1)8d+2Cd for n>2f4+1 
325 +5)(f+1)3d+2Cd for n< 2f. 


The bound of Section 3.5.2 can be slightly modified to give a bound of 


(2V2C +6)(f+1)8d+2Cd for n<2f. 


Using the second timeout task, in which a process waits for d/(jic,) steps between every 
pair of messages, adds a delay of up to C(d/) to every message. Each message is then in 


effect be delayed by a total of up to d(1 + C/,). This gives bounds of 
4A(f+1)A4+ C/pwd+(C?d/u+Cd) for n>2f+1 


and 


B45 +5)(F +0 ct + (C?d/p + Cd) 
and OC ya 1 2/VC +6)(f + 1)d+ (C?d/p + Cd) 


It may be possible that a more clever stategy would allow processes to send messages 
of the algorithm on demand by more closely intertwining the main algorithm and the fault- 


detection task. 
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Chapter 6 


Conclusions 


We first summarize the known bounds for consensus: 


Failure type n> Lower bound Upper bound Reference 


(f eid Xf +l)d+Cd [ADLS90] 


Omissions (sending) Thm. 3.21 


Stopping 


Thm. 3.25 
Thm. 3.28 


“Timing” (see below) 


Byzantine Thm. 4.4 


Auth. Byzantine (see below) 


(see below) 


The bounds for stopping and omission failures (for n > 2f +1) are tight to within approx- 
imately a constant factor (2 and 4, respectively). The bounds for omission failures when 
n <2f are not tight; an improvement in either direction would be interesting. It has been 
noted by Bharali ([B91]) that the running time for omissions failures can be improved to 
3(f + 1)d + Cd by the following modification to the algorithm. The improvement is ob- 
tained by reducing the delay caused by a process that must wait for acknowledgments before 

°This is an updated version of the original Chapter 6. It differs by the inclusion of the following: the 
upper bound for authenticated Byzantine failures when n > 2f +1, the improvement of the constant from 


4 to 3 for the running time of the omissions algorithm ([B91]), and the more careful analysis of the running 
time in the model of [HK89]. 


60 


sending an r message. Recall that if p; sends an r — 1 message at exactly ¢,_;— the latest 
possible time—and immediately thereafter receives an r message, it may have to wait until 
time t,_1 + 2d to receive enough acknowledgments for its r — 1 message before sending an 
r message and advancing to phase r+ 1. Thus a process p; receiving the r message from p; 
would not receive it until t,_; + 3d. The idea is to let p; send a “virtual r message”, even 
though it has not yet received n — f acknowledgments for its r — 1 message. Process p; 
does not treat a virtual r message from p; as a regular r message until it sees that p; has 
received enough acknowledgments for its r — 1 message (recall that all messages, including 
acknowledgments are broadcast to all processes). Thus, if p; does get enough acknowledg- 
ments, then both p; and p; receive them by time t,_, + 2d and p; has effectively received a 


(real) r message from p; by time t,_1 + 2d, saving time d. 


For failures less benign than omissions, this thesis leaves open a large gap in time com- 


plexity. In particular, the following central question remains unanswered: 
Does consensus in the presence of Byzantine failures require time Q(fCd)? 


The difficulty of this problem seems to lie not in the potential of for arbitrary message 
content but in the potential for timing misbehavior. We believe an important step towards 


answering this question will be to obtain tight bounds for “timing failures”, described below. 


Timing failures 


We say that a process suffers a “timing failure” if the time between some pair of successive 
steps is not in the interval [c,,¢2]. The simple direct rounds simulation first described in 
Chapter 3 tolerates timing failures as well, implying a consensus algorithm with running 
time (f +1)(Cd+d). The algorithm of [ADLS90] is also correct despite timing failures, but 
each of its phases may take up to time Cd +d. In fact, no algorithm is known to tolerate 
timing failures in less than time O( fC d). 


Byzantine failures with authentication 


First note that the direct simulations outlined at the beginning of Chapter 3 do not work in 
the presence of Byzantine failures, even with authentication, and our general simulation of 
Chapter 4 itself requires n > 3f +1. 


The upper bound for authenticated Byzantine failures with n > 2f+1 can be obtained by 
a very simple modification of the algorithm for Byzantine failures: change “If |V"| > 2f+1” 
to “relay V"t!” (unconditionally). In other words, to ensure that every process receives f +1 


round r messages (and therefore sends its own round r message), it is sufficient to relay the 
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f +1 round r messages already received—these messages are signed and therefore believable. 


This protocol works for n > 2f +1 and achieves the same time complexity. 


When n < 2f, the only obvious algorithm to tolerate authenticated Byzantine failures 
is a costly simulation of a synchronous algorithm. The simulation requires that processses 
begin synchronized and time out each other’s timeouts by terminating round 7 after (C'~! + 


--+-+C+41)d/cq, steps. 


The lower bound for authenticated Byzantine failures, not presented in this thesis, is 
interesting (greater than Cd) only for the limited range of n < 2f, and therefore says 
nothing interesting about unauthenticated Byzantine failures. The proof of this bound is 
similar to the “shifting scenarios” proofs of [FLM86]. 


Before suggesting other directions for further research, we first comment on the implica- 


tions of our bounds for consensus in a closely related model. 


6.1 Consensus in the related model of [HK89] 


Herzberg and Kutten [HK89] consider a model in which the actual worst-case message delay 
in a given execution, 6, may be much less than the a priori worst-case bound on message 
delay, A. It is thus desirable for the running time of algorithms to depend minimally on A. 
This model raises similar concerns as our model does; in particular, detecting the absence of 


a message may be much more expensive than receiving the message. 


For the consensus problem, it is not difficult to see that direct implementation of syn- 
chronous algorithms gives a running time of O( fA) for any type of failures; on the other 
hand, clearly the synchronous lower bound implies that no algorithm can guarantee a running 
time of less than (f + 1)é. In this model, our algorithms yield an improvement over direct 
simulation strategies similar to the corresponding improvement in our semi-synchronous 


model. 


It is not difficult to see that our algorithms may be run without modification in the model 
of [HK89], yielding the same running times with the syntactic substitution of A for Cd and 
6 for d. Thus we obtain bounds of 


4(f+1)6+A for n>2f4+1 
B25 +5)(f+Ié+A for n< 2f. 


The bound involving VC carries over with ,/A/6 instead of VC, giving a bound of (2,/A/6+ 
6)(f+1)6+A. If 6 < A, these are significant improvements over the bounds obtainable 


by direct simulation. Moreover, it is possible to prove better bound for these algorithms 
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in general, depending on the ratio of A to 6; the bound of 4(f + 1)d+ Cd is realized by 
an execution only if A = 46. This is because process clocks are perfectly synchronized: 
whereas in our model the time between failure and detection may be anywhere in the range 
[d,d+ Cd], in this model it must be A (plus or minus twice the step time, which is assumed 
to be much less than 6; see [ADLS90], §7).) The length of each phase except the last must 
therefore be A and must have at least A/é — 3 failures (processes in B,). There are thus at 
most (f + 1)/(A/é — 3) phases except for the last phase, and the running time is at most 
(AJ +1)6+A. The running time is the maximum of this expression and 4(f + 1)6+A. 
Similarly for the stopping failures algorithm of [ADLS90], the running time is the maximum 
of 2(f + 1)d) + Cd and (SIS +1)6+ A. Our algorithm for Byzantine failures is not 
interesting in this model, as it is trivial to design an algorithm taking only time (f + 1)A 
(our algorithm takes (2f + 1)A + f6). 


In comparison, the algorithms of [DLS88] may also be used in the model of [HK89]. For 
stopping and omission failures (sending and receiving), their algorithms require n > 2f + 1; 
for Byzantine failures, they require n > 3f +1. Their algorithms assume only that an upper 
bound on message delay time exists—it may not be known to the processes; the running time 
is a function of the maximum message delay in the given execution. The running times are 
O(6? +n?) for all types failures. (As noted in [DLS88], the running times can be improved to 
O(6? + f?). We also note that the different model considered there, which enables processes 


to send to at most one process per step, does not affect the time bound asymptotically.) 


Note that in contrast to our algorithm and the algorithm of [ADLS90], the running times 
of the algorithms of [DLS88] in the [HK89] model do not depend at all on A, the upper bound 
on message delay time. This is possible because the model of [HK89] provides an extra degree 
of power to the algorithm by assuming that process clocks are perfectly synchronized. The 
algorithms of [DLS88] do not give good bounds in our model; the running times depend only 
polynomially on the ratio of process step rate, C’. This difference in the model also accounts 
for the simplicity of solving consensus in the presence of Byzantine failures in the [HK89] 


model, relative to our semi-synchronous model. 


6.2. Directions for further research 


There are many possible directions for interesting research addressing the issues and concerns 


of real-time behavior of distributed systems: 


e The existence of the underlying synchronous algorithm described in Section 3.2 suggests 
that the results of [ADLS90] and this thesis may be generalizable to certain classes of 


synchronous algorithms. For instance, the properties of the underlying synchronous 
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algorithm that make it amenable to “efficient” simulation in our model are that it is 
“early-stopping” and that processes advance to further rounds only because of messages 


received (not because of messages omitted). 


— Can the properties sufficient for efficient simulation be clearly characterized? 


— Can these properties be shown necessary by proving lower bounds with large 


dependency on C’ for synchronous algorithms lacking these properties? 


— Are the factors of 4 and /C inherent to such simulations with n < 2f and 


tolerating omission failures? 


What classes of problems are in fact affected by timing uncertainty? Perhaps problems 
solvable in asynchronous systems need not be affected. Can they be helped by timing 


assumptions? Are only fault-tolerant problems affected? 


Similar questions can be asked in the context of the model of Herzberg and Kutten 
([HK89]): What can be said about converting synchronous algorithms with running 
times as a function of message delay d to algorithms that depend on the actual worst- 


case message delay 6 rather than the a priori worst-case message delay A? 


What can be said about simulating synchronous algorithms that do not operate in 


rounds? 


Other work ({S5DC90]) on the real-time complexity of the consensus problem assumes 
a different model of semi-synchrony. There, continuous local clocks are assumed to 
be within a fixed constant € of each other and to stay within a linear envelope of real 
time. Insight into how these two models are related would enable a comparison of the 
bounds that have been obtained. In particular, using the assumptions of our model, 


for what values of their parameters can their model be implementable? 


We have given a straightforward implementation of our consensus algorithm using 
bounded capacity message links. Can a more involved approach avoid merely effectively 


increasing the delay of each message? 


For other problems, can bounded capacity message links be used to control implicitly 
message complexity by causing message inefficiency to be manifested as time ineffi- 


ciency? 
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